{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——Logistic 回归\n",
    "log特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，分别调用缺省参数LogisticRegression、LogisticRegression + GridSearchCV以及LogisticRegressionCV进行参数调优。实际应用中LogisticRegression + GridSearchCV或LogisticRegressionCV任选一个即可。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据 & 数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_log</th>\n",
       "      <th>feat_2_log</th>\n",
       "      <th>feat_3_log</th>\n",
       "      <th>feat_4_log</th>\n",
       "      <th>feat_5_log</th>\n",
       "      <th>feat_6_log</th>\n",
       "      <th>feat_7_log</th>\n",
       "      <th>feat_8_log</th>\n",
       "      <th>feat_9_log</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_85_log</th>\n",
       "      <th>feat_86_log</th>\n",
       "      <th>feat_87_log</th>\n",
       "      <th>feat_88_log</th>\n",
       "      <th>feat_89_log</th>\n",
       "      <th>feat_90_log</th>\n",
       "      <th>feat_91_log</th>\n",
       "      <th>feat_92_log</th>\n",
       "      <th>feat_93_log</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.167949</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.172195</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.159571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.159571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.167949</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.162608</td>\n",
       "      <td>0.649561</td>\n",
       "      <td>0.289065</td>\n",
       "      <td>0.489076</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.165443</td>\n",
       "      <td>0.260365</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.172195</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 95 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_log  feat_2_log  feat_3_log  feat_4_log  feat_5_log  feat_6_log  \\\n",
       "0   1    0.167949         0.0         0.0    0.000000    0.000000    0.000000   \n",
       "1   2    0.000000         0.0         0.0    0.000000    0.000000    0.000000   \n",
       "2   3    0.000000         0.0         0.0    0.000000    0.000000    0.000000   \n",
       "3   4    0.167949         0.0         0.0    0.162608    0.649561    0.289065   \n",
       "4   5    0.000000         0.0         0.0    0.000000    0.000000    0.000000   \n",
       "\n",
       "   feat_7_log  feat_8_log  feat_9_log   ...     feat_85_log  feat_86_log  \\\n",
       "0    0.000000    0.000000         0.0   ...        0.172195     0.000000   \n",
       "1    0.000000    0.159571         0.0   ...        0.000000     0.000000   \n",
       "2    0.000000    0.159571         0.0   ...        0.000000     0.000000   \n",
       "3    0.489076    0.000000         0.0   ...        0.000000     0.165443   \n",
       "4    0.000000    0.000000         0.0   ...        0.172195     0.000000   \n",
       "\n",
       "   feat_87_log  feat_88_log  feat_89_log  feat_90_log  feat_91_log  \\\n",
       "0     0.000000          0.0          0.0     0.000000          0.0   \n",
       "1     0.000000          0.0          0.0     0.000000          0.0   \n",
       "2     0.000000          0.0          0.0     0.000000          0.0   \n",
       "3     0.260365          0.0          0.0     0.000000          0.0   \n",
       "4     0.000000          0.0          0.0     0.142178          0.0   \n",
       "\n",
       "   feat_92_log  feat_93_log   target  \n",
       "0          0.0          0.0  Class_1  \n",
       "1          0.0          0.0  Class_1  \n",
       "2          0.0          0.0  Class_1  \n",
       "3          0.0          0.0  Class_1  \n",
       "4          0.0          0.0  Class_1  \n",
       "\n",
       "[5 rows x 95 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# 请自行在log(x+1)特征和tf_idf特征上尝试，并比较不同特征的结果，\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"Otto_FE_train_log.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 61878 entries, 0 to 61877\n",
      "Data columns (total 95 columns):\n",
      "id             61878 non-null int64\n",
      "feat_1_log     61878 non-null float64\n",
      "feat_2_log     61878 non-null float64\n",
      "feat_3_log     61878 non-null float64\n",
      "feat_4_log     61878 non-null float64\n",
      "feat_5_log     61878 non-null float64\n",
      "feat_6_log     61878 non-null float64\n",
      "feat_7_log     61878 non-null float64\n",
      "feat_8_log     61878 non-null float64\n",
      "feat_9_log     61878 non-null float64\n",
      "feat_10_log    61878 non-null float64\n",
      "feat_11_log    61878 non-null float64\n",
      "feat_12_log    61878 non-null float64\n",
      "feat_13_log    61878 non-null float64\n",
      "feat_14_log    61878 non-null float64\n",
      "feat_15_log    61878 non-null float64\n",
      "feat_16_log    61878 non-null float64\n",
      "feat_17_log    61878 non-null float64\n",
      "feat_18_log    61878 non-null float64\n",
      "feat_19_log    61878 non-null float64\n",
      "feat_20_log    61878 non-null float64\n",
      "feat_21_log    61878 non-null float64\n",
      "feat_22_log    61878 non-null float64\n",
      "feat_23_log    61878 non-null float64\n",
      "feat_24_log    61878 non-null float64\n",
      "feat_25_log    61878 non-null float64\n",
      "feat_26_log    61878 non-null float64\n",
      "feat_27_log    61878 non-null float64\n",
      "feat_28_log    61878 non-null float64\n",
      "feat_29_log    61878 non-null float64\n",
      "feat_30_log    61878 non-null float64\n",
      "feat_31_log    61878 non-null float64\n",
      "feat_32_log    61878 non-null float64\n",
      "feat_33_log    61878 non-null float64\n",
      "feat_34_log    61878 non-null float64\n",
      "feat_35_log    61878 non-null float64\n",
      "feat_36_log    61878 non-null float64\n",
      "feat_37_log    61878 non-null float64\n",
      "feat_38_log    61878 non-null float64\n",
      "feat_39_log    61878 non-null float64\n",
      "feat_40_log    61878 non-null float64\n",
      "feat_41_log    61878 non-null float64\n",
      "feat_42_log    61878 non-null float64\n",
      "feat_43_log    61878 non-null float64\n",
      "feat_44_log    61878 non-null float64\n",
      "feat_45_log    61878 non-null float64\n",
      "feat_46_log    61878 non-null float64\n",
      "feat_47_log    61878 non-null float64\n",
      "feat_48_log    61878 non-null float64\n",
      "feat_49_log    61878 non-null float64\n",
      "feat_50_log    61878 non-null float64\n",
      "feat_51_log    61878 non-null float64\n",
      "feat_52_log    61878 non-null float64\n",
      "feat_53_log    61878 non-null float64\n",
      "feat_54_log    61878 non-null float64\n",
      "feat_55_log    61878 non-null float64\n",
      "feat_56_log    61878 non-null float64\n",
      "feat_57_log    61878 non-null float64\n",
      "feat_58_log    61878 non-null float64\n",
      "feat_59_log    61878 non-null float64\n",
      "feat_60_log    61878 non-null float64\n",
      "feat_61_log    61878 non-null float64\n",
      "feat_62_log    61878 non-null float64\n",
      "feat_63_log    61878 non-null float64\n",
      "feat_64_log    61878 non-null float64\n",
      "feat_65_log    61878 non-null float64\n",
      "feat_66_log    61878 non-null float64\n",
      "feat_67_log    61878 non-null float64\n",
      "feat_68_log    61878 non-null float64\n",
      "feat_69_log    61878 non-null float64\n",
      "feat_70_log    61878 non-null float64\n",
      "feat_71_log    61878 non-null float64\n",
      "feat_72_log    61878 non-null float64\n",
      "feat_73_log    61878 non-null float64\n",
      "feat_74_log    61878 non-null float64\n",
      "feat_75_log    61878 non-null float64\n",
      "feat_76_log    61878 non-null float64\n",
      "feat_77_log    61878 non-null float64\n",
      "feat_78_log    61878 non-null float64\n",
      "feat_79_log    61878 non-null float64\n",
      "feat_80_log    61878 non-null float64\n",
      "feat_81_log    61878 non-null float64\n",
      "feat_82_log    61878 non-null float64\n",
      "feat_83_log    61878 non-null float64\n",
      "feat_84_log    61878 non-null float64\n",
      "feat_85_log    61878 non-null float64\n",
      "feat_86_log    61878 non-null float64\n",
      "feat_87_log    61878 non-null float64\n",
      "feat_88_log    61878 non-null float64\n",
      "feat_89_log    61878 non-null float64\n",
      "feat_90_log    61878 non-null float64\n",
      "feat_91_log    61878 non-null float64\n",
      "feat_92_log    61878 non-null float64\n",
      "feat_93_log    61878 non-null float64\n",
      "target         61878 non-null object\n",
      "dtypes: float64(93), int64(1), object(1)\n",
      "memory usage: 44.8+ MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = train['target']   \n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 默认参数的Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [ 0.68515708  0.6861802   0.68082293]\n",
      "cv logloss is: 0.684053401541\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "#数据集比较大，采用3折交叉验证\n",
    "from sklearn.model_selection import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=3, scoring='neg_log_loss')\n",
    "#%timeit loss_sparse = cross_val_score(lr, X_train_sparse, y_train, cv=3, scoring='neg_log_loss')\n",
    "print ('logloss of each fold is: ',-loss)\n",
    "print ('cv logloss is:', -loss.mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果果然比用原始特征（0.797465616286）好很多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J =  C* sum(logloss(f(xi), yi)) +* penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置参数搜索范围\n",
    "生成GridSearchCV的实例（参数）\n",
    "调用GridSearchCV的fit方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=3, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.67736990448\n",
      "{'penalty': 'l2', 'C': 100}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最佳值在候选参数的边缘，再尝试更大的候选参数，直到找到拐点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VNX9//HXZ7IQEgIEEpYkYJBN\nCJAoUcQNFRBUTOpWa1utVotYa3er31+tFq1tv4/221pbrCvaWutSlwKKgju4Cxo2EQVECAQIiawh\nZDu/P2YIEUIymWS4M5n38/GYR2Y5987nCvLOPfeec8w5h4iICIDP6wJERCRyKBRERKSBQkFERBoo\nFEREpIFCQUREGigURESkgUJBREQaKBRERKSBQkFERBrEe11Aa6Wnp7ucnByvyxARiSqLFy/e5pzL\naKld1IVCTk4OixYt8roMEZGoYmZfBNNO3UciItJAoSAiIg0UCiIi0iDqrimIiOxXU1NDSUkJVVVV\nXpcSMZKSksjOziYhISGk7RUKIhK1SkpKSE1NJScnBzPzuhzPOecoLy+npKSEAQMGhLQPdR+JSNSq\nqqqiZ8+eCoQAM6Nnz55tOnNSKIhIVFMgfFVb/3vETChs3VnFQ299Tl29lh8ViWWX3PsOl9z7jtdl\nRKywhYKZzTSzrWa2/DCfdzOzOWa2xMxWmNmV4aoF4Jv3v8v0OR/z3trycH6NiMSYLl26NDyfPHky\n3bt3Z8qUKU22ve6668jPz2f48OF07tyZ/Px88vPzeeqpp1r1nR9++CEvvvhim+o+nHCeKTwMTG7m\n8+uAj51zecDpwP+ZWWK4iplz/amkJMYxe8mmcH2FiMS4G264gUceeeSwn8+YMYPi4mLmzp3LwIED\nKS4upri4mIsuuqhV3xOVoeCcWwBUNNcESDV/B1iXQNvacNXTOTGOs3L7MHdZKftq68L1NSISw8aP\nH09qampI23722WdMmjSJ0aNHc9ppp/Hpp58C8PjjjzNixAjy8vI444wz2Lt3L7fddhuPPvpoSGcZ\nLfHyltS/AbOBTUAqcIlzrj6cX1iYn8mzH23kjVVlnJXbJ5xfJSJH2PQ5K/h4084W231c6m8TzHWF\n4ZldufW83DbXFoypU6fywAMPMHDgQN566y1+8IMfMH/+fKZPn87rr79O79692b59O507d+aWW25h\n+fLl3Hnnne1eh5ehMAkoBs4EBgIvmdlC59whf6pmNhWYCtC/f/+Qv/CUQen0SElk1pJNCgURiRjb\nt2/n3Xff5cILL2x4r7bW33Fy8sknc/nll3PxxRdzwQUXhL0WL0PhSuD3zjkHrDazz4FjgPcPbuic\nuw+4D6CgoCDk24cS4nycM7IPTy0uYfe+Wrp00tg9kY4i2N/o958hPHHN2HCW0yrOOdLT0ykuLj7k\ns/vvv5/33nuP5557jry8PJYuXRrWWry8JXU9MB7AzHoDQ4G14f7SovwsqmrqeenjzeH+KhGRoKSl\npdG3b1+effZZAOrr61myZAkAa9eu5cQTT+T2228nLS2NjRs3kpqayq5du8JSSzhvSX0MeAcYamYl\nZnaVmU0zs2mBJrcDJ5nZMuAV4Ebn3LZw1bPf6P5pZHXvzKxi3YUkIu3r1FNP5eKLL+aVV14hOzub\nefPmBb3t448/zj333ENeXh65ubk899xzAPzkJz9h5MiRjBw5kgkTJjBixAjOPPNMlixZwrHHHhs9\nF5qdc5e28Pkm4Kxwff/h+HzGeXmZ3L9wLeW799GzS6cjXYKIdCC7d+9ueL5w4cKgtsnJyWH58q8O\n4Tr66KObDJHZs2cf8l5GRkbYFhuLmRHNjRXmZVJX75i7XF1IIrHmiWvGRtT1hEgTk6EwrG8qg3t1\nYXbxRq9LERGJKDEZCmZGUX4mH6z7ko3b93pdjohIxIjJUAA4Ly8TgDma9kJEpEHMhsJRPVPI79dd\ndyGJiDQSs6EAUJSfycrSnXy2JTz3+4pIBHroXP9DmhTToXDuqL74DM2cKiIh2z91dnFxMWPHjiU3\nN5dRo0bxxBNPHNI2GqbOjul5HnqlJnHSwHRmFW/ipxOHaAUnEQlZcnIy//znPxk8eDCbNm1i9OjR\nTJo0ie7duze0mTFjBgDr1q1jypQpTU5rEYwPP/yQ5cuXM3lyc6sThCamzxTAP3Pq+opKijds97oU\nEYliQ4YMYfDgwQBkZmbSq1cvysrKgt5eU2dHiMkj+nDzf5cze8kmju2f5nU5IhKqF26Czctabrc5\nMKFcMNcV+oyEs3/f6lLef/99qqurGThwYNDbaOrsCNE1KYEzhmYwZ0kpN587nDifupBEJHSlpaVc\ndtll/OMf/8DnC64zRlNnR5ii/CzmrdjCO2vKOWVwutfliEgogv2Nfv8ZwpXPt3sJO3fu5Nxzz+U3\nv/kNJ554YtDbaersCHPmMb3o0imeWZr2QkRCVF1dzfnnn9/wW31rxMTU2dEkKSGOSbl9eHHFZqpq\ntH6ziLTek08+yYIFC3j44YcbbjVtzd1FkTJ1tvkXPoseBQUFLhxTxr7xaRnfmfk+93x7NJNHaKlO\nkWiwcuVKhg0b1rqNwth9FCma+u9iZoudcwUtbatrCgEnD+xJz5REZi/ZqFAQ6cg6cBi0B3UfBcTH\n+Zgyqi+vrNzKrqoar8sREfGEQqGRwvxM9tXWM3/FFq9LEZEgRVsXeLi19b+HQqGR4/qnkZ3WmVma\nC0kkKiQlJVFeXq5gCHDOUV5eTlJSUsj70DWFRsyMwrxM7l2wlm2795Gu9ZtFIlp2djYlJSWtmk6i\no0tKSiI7Ozvk7RUKBynKz+Lu19cwd1kpl4/N8bocEWlGQkICAwYM8LqMDkXdRwcZ2ieVob1TtfiO\niMQkhUITCvMzWfzFl2yoqPS6FBGRI0qh0ITC/es3L9XZgojEFoVCE/r1SOa4/t2ZrS4kEYkxCoXD\nKMrP4pPNu/hk806vSxEROWLCFgpmNtPMtprZ8mbanG5mxWa2wszeCFctoThnZF/ifKazBRGJKeE8\nU3gYOOwCombWHbgbKHTO5QKtm2s2zDJSO3HyoHRmL9mkgTEiEjPCFgrOuQVARTNNvgk845xbH2i/\nNVy1hKowL5OSL/fy4Xqt3ywiscHLawpDgDQze93MFpvZ5YdraGZTzWyRmS06kiMXJ+X2JjHex2wt\nviMiMcLLUIgHRgPnApOAX5nZkKYaOufuc84VOOcKMjIyjliBqUkJTBjWi+eXlVJbV3/EvldExCte\nhkIJ8KJzbo9zbhuwAMjzsJ4mFeZlsm13NW+vKfe6FBGRsPMyFGYBp5pZvJklA2OAlR7W06TTh/Yi\ntVO8pr0QkZgQzltSHwPeAYaaWYmZXWVm08xsGoBzbiXwIrAUeB94wDl32NtXvZKUEMfkEX2Yp/Wb\nRSQGhG2WVOfcpUG0+QPwh3DV0F6K8rP4z+ISXvtkK2eP7Ot1OSIiYaMRzUEYO7An6V06qQtJRDo8\nhUIQ4nzGlFF9eXXVVnZq/WYR6cAUCkEqys+kuraeecs3e12KiEjYKBSClN+vO/17JDNb6zeLSAem\nUAjS/vWb31q9ja27qrwuR0QkLBQKrVCUn0m9g7lLS70uRUQkLBQKrTC4dyrD+nZllrqQRKSDUii0\nUmFeJh+t3876cq3fLCIdj0Khlc7L8w9em71EM6eKSMejUGil7LRkjs9JY1axFt8RkY4ndkLBOVj3\nVrvsqjAvk8+27uaTzbvaZX8iIpEidkLhrwXw8DntEgz712/WtBci0tHETihMWwipmTD/l1DftgVz\nenbpxKmD05mzZBP19epCEpGOI3ZCITEZxt8Cmz6C5U+3eXdF+Zls3L6XD9d/2Q7FiYhEhtgJBYBR\nl0CfUfDKdKjZ26ZdTRzeh07xPnUhiUiHEluh4PPBWb+BHRvgvXvatKsuneKZMLw3zy8rpUbrN4tI\nBxFboQBw9DgYMhkW/gn2bGvTroryMqnYU81bq9u2HxGRSBF7oQAw8Tao3gOv/75Nuxk3NIOuSfHM\nVheSiHQQsRkKGUNh9BWwaCZs+yzk3XSKj+PsEX2Zt2Ize6u1frOIRL/YDAWA0/8HEpLhpVvbtJui\n/Ez2VNfx6idb26kwERHvxG4odMmAU34Mq56HdW+GvJsxR/ekV2onZhVrLiQRiX6xGwoAY6+Drlkw\nL/QBbf71mzN5fVUZO/Zq/WYRiW6xHQoJnf0D2kqLYdl/Qt5NUX4m1XVav1lEol9shwLAyK9D3zx4\n5baQB7SNyu5GTs9kZmk6bRGJcgqF/QPadpbAu38PaRf7129+e005W3dq/WYRiV5hCwUzm2lmW81s\neQvtjjezOjO7KFy1tGjAaTDkbP+Att1lIe2iMD8T52CO1m8WkSgWzjOFh4HJzTUwszjgf4F5Yawj\nOBNvg5pKeCO0AW2DeqWSm9mV2Vq/WUSiWNhCwTm3AKhoodn1wNOA9zf5ZwyBgith0UNQtiqkXRTl\nZ7Jkw3bWbdvTzsWJiBwZnl1TMLMs4HygbTPTtadxN7VpQNuUUZkAOlsQkajl5YXmO4EbnXMtzg9h\nZlPNbJGZLSorC63PPyhdMuDUn8KnL8DnC1q9eWb3zpwwoAezijdq/WYRiUpehkIB8LiZrQMuAu42\ns6811dA5d59zrsA5V5CRkRHeqk68Frr1g/k3hzSgrSg/kzVle/i4dGcYihMRCS/PQsE5N8A5l+Oc\nywGeAr7vnPuvV/U0aBjQtgSWPdnqzc8Z0Zd4n2nmVBGJSuG8JfUx4B1gqJmVmNlVZjbNzKaF6zvb\nzYiLoG9+SAPa0lISOW1IBrO1frOIRKFw3n10qXOur3MuwTmX7Zx70Dl3j3PukAvLzrkrnHNPhauW\nVvP5YNIdsHMjvDOj1ZsX5WdSuqOKRV9o/WYRiS4a0Xw4OafA0HPhzT/D7tbdMTthWG+SEnyaOVVE\noo5CoTkTp0NtFbz+u1ZtltIpnonD+zBX6zeLSJRRKDQnfTAUfBcW/wO2ftKqTYvyMvmysoY3P9P6\nzSISPRQKLRl3IySmwMutG9B22pAMunVOUBeSiEQVhUJLUtIDA9pehLVvBL1ZYryPc0b2Yf7HW6is\nrg1jgSIi7UehEIwx+we0tW6FtsK8LCqr63h5pfdTO4mIBKPVoWBmPjPrGo5iIlZCEoy/FTYvg6WP\nB73ZCQN60KdrkgayiUjUCCoUzOzfZtbVzFKAj4FVZnZDeEuLMCMuhMxj4ZXboboyqE386zf35Y1P\nt7K9sjrMBYqItF2wZwrDnXM7ga8Bc4H+wGVhqyoS+Xxw1h2waxO8G/yAtqL8LGrqHC9o/WYRiQLB\nhkKCmSXgD4VZzrkaIPbmcMg5GY6ZAm/eCbu2BLXJiKyuHJ2eoi4kEYkKwYbCvcA6IAVYYGZHAbE5\nDeiE1g1oMzMK8zN59/NyNu/Q+s0iEtmCCgXn3F3OuSzn3DnO7wvgjDDXFpnSB0HBVfBh8APaCvP8\n6zc/t1RnCyIS2YK90PyjwIVmM7MHzexD4Mww1xa5xt0Iianw0q+Can50RhdGZnVjlrqQRCTCBdt9\n9N3AheazgAzgSiC0Fe47gpSecNrP4LP5sOa1oDYpys9k2cYdrC3bHebiRERCF2woWODnOcBDzrkl\njd6LTSdcA936w/xfQX2LK4oyZVQmZlq/WUQiW7ChsNjM5uMPhXlmlgrE9vSfCUkw4VbYsgyWtDyg\nrU+3JMYM6MHs4k1av1lEIlawoXAVcBNwvHOuEkjE34UU20ZcCFmj4dXgBrQV5WexdtseVmyKzRu3\nRCTyBXv3UT2QDdxsZn8ETnLOLQ1rZdHALDCgrRTe+VuLzc8e0YeEONPMqSISsYK9++j3wI/wT3Hx\nMfBDM2vdyjMd1VFjYdh5QQ1o656cyLghGcxZUqr1m0UkIgXbfXQOMNE5N9M5NxOYDJwbvrKizITp\nULcPXrujxaaF+Vls3lnF++sqjkBhIiKt05pZUrs3et6tvQuJaj0HwvFXw0ePwJaPm206YVgvkhPj\nNGZBRCJSsKHwO+AjM3vYzP4BLAZ+G76yolDDgLZbmm2WnBjPxOG9mbuslOra2L6BS0QiT7AXmh8D\nTgSeCTzGOueCX1ggFiT3gNN+DqtfgjWvNtu0KD+THXtrWPBp2REqTkQkOM2Ggpkdt/8B9AVKgA1A\nZuA9aWzMNdC95QFtpw7OIC05QQPZRCTixLfw+f8185kjluc/akp8J5jwa3jqu7DkMTj22002S4jz\ncc7Ivjzz4Ub27KslpVNLfwwiIkdGs2cKzrkzmnkoEJqSewFkFQRWaNtz2GaFeZnsranj5ZXBrcsg\nInIkBDtO4YImHuPNrFcz28w0s61mtvwwn3/LzJYGHm+bWV6oBxFRzGDSHbB7M7x9+AFtx+f0oG83\nrd8sIpGlNdNcPAB8K/C4H/gp8JaZHW5Zzofxj2c4nM+Bcc65UcDtwH1B1hL5+p8Iwwrhrb/ArqaX\n4fT5jMK8TN74tIwv92j9ZhGJDMGGQj0wzDl3oXPuQmA4sA8YA9zY1AbOuQXAYUdoOefeds59GXj5\nLv5pNDqOCb+GuupmB7Sdl5dJbb1j7vLSI1aWiEhzgg2FHOdc487vrcAQ51wFUNMOdVwFvHC4D81s\nqpktMrNFZWVRchtnz4Fwwvfgo3/BlhVNNsnN7MrAjBQNZBORiBFsKCw0s+fM7Dtm9h1gNv61mlOA\n7W0pwMzOwB8KTZ5xADjn7nPOFTjnCjIyMtrydUfWaTdAp1T/LapNMDOK8rP4YF0Fm7bvPcLFiYgc\nKthQuA54CMgHjgX+AVznnNvjnAt5rWYzG4X/WkWRc6481P1ErOQecNovYM0rsPrlJpto/WYRiSTB\njmh2wJvAq8DLwALXxpVizKw//tHRlznnPm3LviLaCd+D7kfB/FuaHNCWk55CXrbWbxaRyBDsLalf\nB94HLgK+DrxnZhe1sM1jwDvAUDMrMbOrzGyamU0LNLkF6AncbWbFZrYo5KOIZPsHtG1dAcWPNtmk\nMD+LFZt2snqr1m8WEW9ZML/wm9kS/FNnbw28zgBeds4d8bEFBQUFbtGiKMsP5+DBibB9A1y/GDp1\n+crHW3dWMeZ3r3D9mYP56cQhHhUpIh2ZmS12zhW01C7Yawq+/YEQUN6KbWX/Cm27N8Pbfz3k415d\nkxh7dE9mF2/U+s0i4qlg/2F/0czmmdkVZnYF8DwwN3xldUD9x8DwInj7Lth56LiEovxM1pVXsrRk\nhwfFiYj4BXuh+Qb8I45HAXnAfc65w95CKocx4ddQV9PkgLbJuX1JjPNp5lQR8VTQXUDOuaedcz91\nzv3EOfdsOIvqsHocDSdM9Q9o2/zVKaG6JSdw+tAM5izZRJ3WbxYRj7S0nsIuM9vZxGOXme08UkV2\nKKf9HJK6wUuHDmgrzM9k6659vLe24w3ZEJHo0NLU2anOua5NPFKdc12PVJEdSnIPGPcL/+psn311\nQNv4Y3qTkhinLiQR8YzuIPLC8VdDWg7Mvxnqahve7pwYx6TcPsxdVsq+2sOv3CYiEi4KBS/Ed4IJ\n06Fs5SED2s7Lz2RnVS1vrIqSif9EpENRKHhleBH0G+O/E2nfgZHMpwxKp0dKorqQRMQTCgWvNAxo\n2+IfuxCQEOfj3JF9eXnlFnbvq21mByIi7U+h4KV+x0Pu+fDWXbDzwJlBUX4mVTX1vPRx06u2iYiE\ni0LBa+NvBVcHrx4Y0HZc/zSyunfWzKkicsQpFLzWY4B/QFvxo7B5GeBfv/m8vEwWfraN8t37PC5Q\nRGKJQiES7B/QNv9m/4yq+LuQ6uodc5erC0lEjhyFQiTonAbjboS1rzes0HZMn1QG9+rC7OKN3tYm\nIjFFoRApjr8a0gY0DGjzr9+cyQfrvmSj1m8WkSNEoRAp4hNh4nQo+wQ+egSAwrwsAOZozIKIHCEK\nhUgyrBD6nQiv/Rb27aJ/z2SO7d9ddyGJyBGjUIgkZjDpDtizFd76CwCFeZmsLN3JZ1t2eVyciMQC\nhUKkyS6A3Avg7b/Bjo2cO6ovPkPTXojIEaFQiEQTAgPaXruDXqlJnDwonVnFm7R+s4iEnUIhEqXl\nwJhroPjfULqU8/IyWV9RSfGG7V5XJiIdnEIhUp36M+jcHeb/ksm5vUmM1/rNIhJ+CoVI1TkNxt0E\nny+g64bXOHNoL+YsKdX6zSISVgqFSFbwXehxNMz/FV8b1Yttu/fxzhqt3ywi4RO2UDCzmWa21cyW\nH+ZzM7O7zGy1mS01s+PCVUvUik+EibfBtlWMr5pPl07xzNK0FyISRuE8U3gYmNzM52cDgwOPqcDf\nw1hL9DpmCvQfS8KC31F4TCovrthMVY3WbxaR8AhbKDjnFgAVzTQpAv7p/N4FuptZ33DVE7X2r9C2\np4xp8XPYVVXL61q/WUTCxMtrClnAhkavSwLvycGyR8OIi+i3aibDU3Yxe4m6kEQkPLwMBWvivSZv\nrTGzqWa2yMwWlZXF6G/J42/BnOO33Z7llZVb2VVV43VFItIBeRkKJUC/Rq+zgSZvxHfO3eecK3DO\nFWRkZByR4iJO2lFw4jTyKuYxqG4N81ds8boiEemAvAyF2cDlgbuQTgR2OOdKPawn8p3yU+icxm2d\nH+PJD9ZTrzELItLOwnlL6mPAO8BQMysxs6vMbJqZTQs0mQusBVYD9wPfD1ctHUbn7tjpNzG6fjkp\n61/h+sc/0p1IItKuLNomWSsoKHCLFi3yugzv1NXgZoxhd2Ul5+/4CT1yRnH/ZQV0S07wujIRiWBm\nttg5V9BSO41ojjZxCdgF95MaV8uLKb8mY8M8Lr73bTZpyU4RaQcKhWiUPRqmvkF872HMiP8zF21/\niItmLOSTzTu9rkxEopxCIVp1y4IrX4DjLmeqPcsfa3/LVfe8pLmRRKRNFArRLL4TFP4VptzJWFvO\nf+z/ccfMp5ijKbZFJEQKhY6g4Ersyrn0SXY8lfgrXnzi7zywcK3XVYlIFFIodBT9TsA3bQGJWXnM\nSLyL2nm/4jdzlmosg4i0ikKhI0ntg++K56kvuIpp8c8x7v1ruenR19lXq7EMIhIchUJHE5+Ib8qf\ncIV/5aT4VVz/2fe45Z5/s2Ov5koSkZYpFDooO+5y4q6aR3pKHNPLfsI9d91B6Q6NZRCR5ikUOrLs\n0XS+7k329T6WG/f+mTfuuppVG5tb4kJEYp1CoaPrkkG3a+ZSMeIqvlH3HDvvP5dFK1Z5XZWIRCiF\nQiyIS6DHRX+iYtLfGMVqsp6czMI35ntdlYhEIIVCDOkx9jKqL59LXFw8J7z6Td544k6vSxKRCKNQ\niDGpRx9P1x+9xefJIxm38lYWzbiS+pp9XpclIhFCoRCDkrr1YvDP5vNWr29SUPYMn//fmezbrqkx\nREShELPi4hM46dq7eXn478jc+yl77jqF3avf8bosEfGYQiGGmRkTvv593j3zCXbXxdHpX1PYvvA+\nr8sSEQ8pFIQzxp1J6ddf4D2XS/dXbmD7E9dCra4ziMQihYIAMCZ3ED2+N4uHfBfQfeW/2XXPJNip\n6wwisUahIA2GZ6cx8foZ/DrpRnxlH1M141T4QtcZRGKJQkG+IjstmR//8Of8Mv0vlO6Np/7hKfD+\n/eA0BbdILFAoyCG6Jyfy+2lf566j7+O12pEw9+e4/34faqq8Lk1EwkyhIE1KSojjj5efxsLRd/GX\n2guwJf+mfuZk2FHidWkiEkYKBTmsOJ9xa9FIOk28maurf0bV5lXU3zsOPl/odWkiEiYKBWmWmTFt\n3ECmXHwV51ffRklVEu6fRfDO3brOINIBKRQkKF87NotfXXE+F9f9hgU2Gub9DzwzFaorvS5NRNpR\nWEPBzCab2SozW21mNzXxeX8ze83MPjKzpWZ2TjjrkbY5ZXA6M685kxt8N/BXLsEt+w/MPAu+/MLr\n0kSknYQtFMwsDpgBnA0MBy41s+EHNbsZeNI5dyzwDeDucNUj7SM3sxtPf/8U/pv6TabW/oKa8nVw\n3zhY86rXpYlIOwjnmcIJwGrn3FrnXDXwOFB0UBsHdA087wZoCG0U6NcjmaevPYkvs05n4u7pfOnr\nAf+6EN76i64ziES5cIZCFrCh0euSwHuN/Rr4tpmVAHOB65vakZlNNbNFZraorKwsHLVKK3VPTuRf\nV49h6PA8Ti7/JSu7nw4v3QJPXQnVe7wuT0RCFM5QsCbeO/jXyEuBh51z2cA5wCNmdkhNzrn7nHMF\nzrmCjIyMMJQqoUhKiOPub43morFDObv0KmalX4P7eBY8MBEq1npdnoiEIJyhUAL0a/Q6m0O7h64C\nngRwzr0DJAHpYaxJ2lmcz5hemMuNk4fxo5Jx/L7Hb3A7N8J9p8NnL3tdnoi0UjhD4QNgsJkNMLNE\n/BeSZx/UZj0wHsDMhuEPBfUPRRkz49rTB/Knr+fxYOkArkz4AzWp2fDoRbDgj7rOIBJFwhYKzrla\n4AfAPGAl/ruMVpjZbWZWGGj2M+B7ZrYEeAy4wjn9CxKtLjgum4euPJ4PdnTlrB03s2tQIbx6Ozx5\nGezb5XV5IhIEi7Z/gwsKCtyiRYu8LkOasXzjDq58+AP21dQyp2AZRy3+HfQcBN94FNIHe12eSEwy\ns8XOuYKW2mlEs7S7EVndeObak0hPTWLiuyN495QHoXIb3H8mrHrB6/JEpBkKBQmLfj2SeXraSYzM\n6salLyfyn+P+BT0GwGPfgNd/D/X1XpcoIk1QKEjYpKUk8ujVY5g4rDc3vPwlf8j6C27UN+D138Hj\n34SqHV6XKCIHUShIWCUlxPH3b4/mshOPYsabm/jxvmuonfS/sPolf3fS1k+8LlFEGlEoSNjF+Yzb\ninK5YdJQZi0p5Tsr8tlz6bP+M4UHxsMrt8PHs6B8jbqVRDwW73UBEhvMjOvOGESfrknc+PRSLtzd\nhUe+OZ+Mf54OC/94oGFCMvQaDn1GQO8R0DvX/0jq5lXpIjFFoSBH1IWjs8lI7cS1/1rM1x5Zx8Pf\nXcHgtDgo+wS2LIctK/yPFf+FxQ8f2LBbf3849NkfFCOgx9Hgi/PsWEQ6Io1TEE8cGMtQxw/HD2ZA\negpH9UwmOy2ZpIQ4/yjonZsCIbHsQFhs+wxcnX8n8Z2h17BAWIw8cFbROc3bgxOJQMGOU1AoiGc2\nVFRy1p8XsLem7ivv9+maRP+MGVtzAAAJIUlEQVQeyfTrkcxRPZPp3yOZ/oGfPTvVY2WrGp1VLIfN\ny2FvxYEddM0+EBD7u6F6DIQ4nRhL7FIoSFRwzlG+p5r1FZWsL69kfUUlX5RXsqHC/3zzzqqvtE9J\njPtqWAQeAzrtou++NSSUrWh0VvEp1Nf6N4xPgoxj/AHRuAsquYcHRy1y5CkUpEOoqqmj5Et/UBwc\nGOsrKtlXe+BuJZ9B326dGwIjJy2e4fGl5NSto/eez+hUsdJ/VlG57cAXpGY2OqsIdEH1HARxCR4c\nrUj4BBsKOp+WiJaUEMegXqkM6pV6yGf19Y6y3fsaAmN9+R5/cFRU8vLKLWzbXR1o2QfoQ9ekMziq\nZwojeleR36mEIawna98aum//lIS1r2P1Nf7mcYkHzioad0GlaFZ36fh0piAd1p59tQ1nFA1dUxX+\nM42SLyupqTvwd7+zr46x3co5vvMmRsRtIKfuczIq15BU1Wgm9y69G90mG+iG6jkY4hM9ODqR1tGZ\ngsS8lE7xDOvblWF9ux7yWV29o3TH3kMC44WKwdyzrZIde/1nDT3ZwTG+9YzutIn8mhKGlHxBn7UL\niHf+z50vATKGYL1H+u+ESuoKvgT/2UZcQuCRCL74Jt5LOMzrQDvdbiseUChITIrzGdlp/ltgTxp4\n6Oc7KmsOnGVUVLK+Yg8PBq5pbK3aRQ6lHGPrGeZbT27peoZvnU+6e7xda6zHR73FU+dLoN7i/Q9f\nPPWWcOA9XwIu8F69Lx7nSwz8TPB/ZvG4uMTAzwScL/AIPMeXgIvzb7c/lFxcIs4XxyEr6trBK+x+\n9bVr/PqQxXhb2LaFfbdUy6H9HU2tBhykEHpPQvu21n9P98xBHDU0P6RvC5ZCQaQJ3ZITGJncjZHZ\nh46krqmrZ+OXexsC482KSv5dXklFRRlxtXvxuVriXA1xrpa4ev9Pn6sl3tXiY/9ndcS7mob3/M/r\n/J9RR7yrJZ7AI/A8gVoSqCPBaomnjkT8PxOoJdH2NDxPCPyMp45EO/C68Wc+i65uY/F71R3HUdNf\nC+t3KBREWikhzkdOego56SlH7DudczgH9c5RH/h54LX/Pdfos8af1zjYV39Q+7o6qNuHq6vF1Vbj\n6quhtgZXV42rq4G6mkPOA2jutWvms4Ne2yG/iR+8rxY+P2R/NPs5zmG+1v0u70L63T+EbQ45Q2pe\nbka/lhu1kUJBJAqYGWbga0u3iEgQNEuqiIg0UCiIiEgDhYKIiDRQKIiISAOFgoiINFAoiIhIA4WC\niIg0UCiIiEiDqJsl1czKgC9C3Dwd2NZiq+igY4lMHeVYOspxgI5lv6OccxktNYq6UGgLM1sUzNSx\n0UDHEpk6yrF0lOMAHUtrqftIREQaKBRERKRBrIXCfV4X0I50LJGpoxxLRzkO0LG0SkxdUxARkebF\n2pmCiIg0I+ZCwcxuN7OlZlZsZvPNLNPrmkJlZn8ws08Cx/OsmXX3uqZQmdnFZrbCzOrNLOruFDGz\nyWa2ysxWm9lNXtcTKjObaWZbzWy517W0lZn1M7PXzGxl4O/Wj7yuKRRmlmRm75vZksBxTA/r98Va\n95GZdXXO7Qw8/yEw3Dk3zeOyQmJmZwGvOudqzex/AZxzN3pcVkjMbBhQD9wL/Nw5t8jjkoJmZnHA\np8BEoAT4ALjUOfexp4WFwMxOA3YD/3TOjfC6nrYws75AX+fch2aWCiwGvhZtfy5mZkCKc263mSUA\nbwI/cs69G47vi7kzhf2BEJBCKKtnRwjn3HznXG3g5btAtpf1tIVzbqVzbpXXdYToBGC1c26tc64a\neBwo8rimkDjnFgAVXtfRHpxzpc65DwPPdwErgSxvq2o957c78DIh8Ajbv1sxFwoAZnaHmW0AvgXc\n4nU97eS7wAteFxGjsoANjV6XEIX/+HRkZpYDHAu8520loTGzODMrBrYCLznnwnYcHTIUzOxlM1ve\nxKMIwDn3S+dcP+BR4AfeVtu8lo4l0OaXQC3+44lYwRxLlGpq4eSoPQPtaMysC/A08OODegqihnOu\nzjmXj7834AQzC1vXXny4duwl59yEIJv+G3geuDWM5bRJS8diZt8BpgDjXYRfIGrFn0u0KQH6NXqd\nDWzyqBZpJNAH/zTwqHPuGa/raSvn3HYzex2YDITlZoAOeabQHDMb3OhlIfCJV7W0lZlNBm4ECp1z\nlV7XE8M+AAab2QAzSwS+Acz2uKaYF7hA+yCw0jn3J6/rCZWZZey/s9DMOgMTCOO/W7F499HTwFD8\nd7p8AUxzzm30tqrQmNlqoBNQHnjr3Si+k+p84K9ABrAdKHbOTfK2quCZ2TnAnUAcMNM5d4fHJYXE\nzB4DTsc/G+cW4Fbn3IOeFhUiMzsFWAgsw///O8D/c87N9a6q1jOzUcA/8P/d8gFPOuduC9v3xVoo\niIjI4cVc95GIiByeQkFERBooFEREpIFCQUREGigURESkgUJBpAlmtrvlVs1u/5SZHR143sXM7jWz\nNYFZLheY2RgzSww875CDSCU6KRRE2pmZ5QJxzrm1gbcewD/J3GDnXC5wBZAemDzvFeASTwoVaYJC\nQaQZ5veHwBxNy8zsksD7PjO7O/Cb/3NmNtfMLgps9i1gVqDdQGAMcLNzrh4gMJvq84G2/w20F4kI\nOm0Vad4FQD6Qh3+U7wdmtgA4GcgBRgK98E/LPDOwzcnAY4HnufhHZ9cdZv/LgePDUrlICHSmINK8\nU4DHArNUbgHewP+P+CnAf5xz9c65zcBrjbbpC5QFs/NAWFQHFoER8ZxCQaR5TU2L3dz7AHuBpMDz\nFUCemTX3/1onoCqE2kTanUJBpHkLgEsCi5xkAKcB7+NfEvHCwLWF3vgnkdtvJTAIwDm3BlgETA/M\n2omZDd6/hoSZ9QTKnHM1R+qARJqjUBBp3rPAUmAJ8Crwi0B30dP411FYjn9d6feAHYFtnuerIXE1\n0AdYbWbLgPs5sN7CGUBUzdopHZtmSRUJkZl1CSym3hP/2cPJzrnNgTnvXwu8PtwF5v37eAb4nyhe\nn1o6GN19JBK65wKLnyQCtwfOIHDO7TWzW/Gv07z+cBsHFuT5rwJBIonOFEREpIGuKYiISAOFgoiI\nNFAoiIhIA4WCiIg0UCiIiEgDhYKIiDT4/zMMU49uqBivAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a13459a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#pd.DataFrame(grid.cv_results_).to_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#cvresult = pd.DataFrame.from_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#test_means = cv_results['mean_test_score']\n",
    "#test_stds = cv_results['std_test_score'] \n",
    "#train_means = cvresult['mean_train_score']\n",
    "#train_stds = cvresult['std_train_score'] \n",
    "\n",
    "\n",
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    #plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'logloss' )\n",
    "plt.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的logloss。可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上当C=100时性能最好（L1正则）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用LogisticRegressionCV实现正则化的 Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "           class_weight=None, cv=3, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1.0, max_iter=100, multi_class='ovr',\n",
       "           n_jobs=1, penalty='l1', random_state=None, refit=True,\n",
       "           scoring='neg_log_loss', solver='liblinear', tol=0.0001,\n",
       "           verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]\n",
    "#nCs = 9  #Cs values are chosen in a logarithmic scale between 1e-4 and 1e4.\n",
    "\n",
    "# 大量样本（6W+）、高维度（93），L1正则 --> 可选用saga优化求解器(0.19版本新功能)\n",
    "# LogisticRegressionCV比GridSearchCV快\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 3, scoring='neg_log_loss', penalty='l1', solver='liblinear', multi_class='ovr')\n",
    "lrcv_L1.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Class_1': array([[-0.14540941, -0.13888327, -0.11033725, -0.08749023, -0.08507874,\n",
       "         -0.08521415, -0.08524412],\n",
       "        [-0.14541674, -0.13889324, -0.11085722, -0.08637617, -0.0836447 ,\n",
       "         -0.08395953, -0.08400906],\n",
       "        [-0.14542407, -0.1389033 , -0.10961539, -0.08513374, -0.08210621,\n",
       "         -0.08207292, -0.08208421]]),\n",
       " 'Class_2': array([[-0.57507308, -0.45068348, -0.34931526, -0.32030421, -0.31669393,\n",
       "         -0.31659969, -0.31659842],\n",
       "        [-0.5750947 , -0.45161968, -0.34949774, -0.32071066, -0.31833695,\n",
       "         -0.31854972, -0.31858148],\n",
       "        [-0.57511632, -0.44816622, -0.34621376, -0.31954067, -0.31724365,\n",
       "         -0.31728877, -0.31729992]]),\n",
       " 'Class_3': array([[-0.38749796, -0.34550656, -0.28295377, -0.26357786, -0.26132108,\n",
       "         -0.26135071, -0.26136374],\n",
       "        [-0.38751797, -0.34476714, -0.2853468 , -0.26844802, -0.2666547 ,\n",
       "         -0.26663302, -0.26663829],\n",
       "        [-0.38753799, -0.345958  , -0.28507455, -0.26546101, -0.26289776,\n",
       "         -0.26297127, -0.26298316]]),\n",
       " 'Class_4': array([[-0.18415472, -0.17893305, -0.1464004 , -0.13108627, -0.12861342,\n",
       "         -0.12885358, -0.12891715],\n",
       "        [-0.18416452, -0.17894572, -0.14574117, -0.12915447, -0.12653613,\n",
       "         -0.12661015, -0.12666254],\n",
       "        [-0.18417431, -0.1789584 , -0.14722002, -0.13256127, -0.13008921,\n",
       "         -0.1301892 , -0.13022441]]),\n",
       " 'Class_5': array([[-0.18648008, -0.07880459, -0.03785669, -0.01959767, -0.01458579,\n",
       "         -0.01422857, -0.01448869],\n",
       "        [-0.18649002, -0.08068992, -0.04168659, -0.02229938, -0.01630894,\n",
       "         -0.01630374, -0.01660148],\n",
       "        [-0.18649996, -0.08324412, -0.04253587, -0.02253446, -0.01632   ,\n",
       "         -0.01552271, -0.01570203]]),\n",
       " 'Class_6': array([[-0.53896138, -0.22606402, -0.13831651, -0.11811352, -0.1183933 ,\n",
       "         -0.11870177, -0.11873754],\n",
       "        [-0.53898383, -0.22334704, -0.13362228, -0.10875925, -0.10683454,\n",
       "         -0.10683099, -0.10683452],\n",
       "        [-0.53895681, -0.22146801, -0.13126193, -0.10825376, -0.10690238,\n",
       "         -0.10708653, -0.10711044]]),\n",
       " 'Class_7': array([[-0.19136209, -0.17442254, -0.1274818 , -0.10595623, -0.10419447,\n",
       "         -0.10434829, -0.10437998],\n",
       "        [-0.19125602, -0.1737928 , -0.1283954 , -0.10997782, -0.10838408,\n",
       "         -0.10855415, -0.10857447],\n",
       "        [-0.19126625, -0.17360164, -0.12434223, -0.10261606, -0.09992786,\n",
       "         -0.09999191, -0.10001154]]),\n",
       " 'Class_8': array([[-0.40138531, -0.30024447, -0.136595  , -0.10852201, -0.10712896,\n",
       "         -0.10736753, -0.10739802],\n",
       "        [-0.40133013, -0.30120056, -0.13294125, -0.10269237, -0.10081911,\n",
       "         -0.1009001 , -0.10091663],\n",
       "        [-0.40135062, -0.29938001, -0.13476343, -0.10676062, -0.1048938 ,\n",
       "         -0.10500785, -0.10502702]]),\n",
       " 'Class_9': array([[-0.28236401, -0.2212519 , -0.11548552, -0.09069683, -0.08833885,\n",
       "         -0.08879295, -0.08887426],\n",
       "        [-0.2823794 , -0.22325211, -0.11742408, -0.09472103, -0.09206397,\n",
       "         -0.09216805, -0.09220315],\n",
       "        [-0.28229682, -0.22399186, -0.11796379, -0.09341746, -0.09053635,\n",
       "         -0.09056858, -0.09058582]])}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4FeX5//H3nZ0dgYCssi8REGpw\nQ3FXcEFcWrW1ta39Uq3+bGtVRG296r50/9a22lrbfqtSFVFaQdwAV5QgyBaWACIBkaDsS0LI/fvj\nDPYIgZzkZDInyed1XefyzDPPzLmnFT7O9jzm7oiIiNRUWtQFiIhI/aYgERGRpChIREQkKQoSERFJ\nioJERESSoiAREZGkKEhERCQpChIREUmKgkRERJKSEXUBdaFdu3bevXv3qMsQEalX5syZs9Hdc6vq\n1yiCpHv37hQUFERdhohIvWJmqxPpp0tbIiKSlFCDxMxGmtlSMysys1sqWX+1mS0ws3lm9paZ5QXt\nZ5rZnGDdHDM7LW6bGcE+5wWf9mEeg4iIHFpol7bMLB14GDgTKAZmm9lkd18c1+1Jd/9T0H808Ctg\nJLARON/d15nZQGAa0Dluu2+4u65ViYikgDDPSI4Bitx9pbuXAROAC+I7uPvWuMVmgAftc919XdC+\nCMgxs+wQaxURkRoKM0g6A2vilov58lkFAGZ2rZmtAB4Erq9kPxcDc929NK7t8eCy1k/NzCr7cTMb\na2YFZlZQUlJS86MQEZFDCjNIKvsL/oBZtNz9YXfvBYwDbv/SDsyOBB4Avh/X/A13HwScFHy+WdmP\nu/uj7p7v7vm5uVU+vSYiIjUUZpAUA13jlrsA6w7SF2KXvsbsWzCzLsAk4FvuvmJfu7uvDf65DXiS\n2CU0ERGJSJhBMhvoY2Y9zCwLuAyYHN/BzPrELZ4LLA/aWwMvAuPd/e24/hlm1i74ngmcBywM6wBe\nWrieJ9/7OKzdi4g0CKE9teXu5WZ2HbEnrtKBv7r7IjO7Eyhw98nAdWZ2BrAH2ARcGWx+HdAb+KmZ\n/TRoOwvYAUwLQiQdeBX4c1jH8PzctbyxvIQz8trTvkVOWD8jIlKvmfsBty0anPz8fK/Jm+2rNu7g\nzF/N5GvDunLvhYNCqExEJHWZ2Rx3z6+qn95sP4Qe7ZpxxXFH8K/Za1j+6baoyxERSUkKkipcf3of\nmmamc//UJVGXIiKSkhQkVWjTLIsfnNqb15Zs4J0VG6MuR0Qk5ShIEvCd4d3p1CqHe6cUUlHR8O8p\niYhUh4IkATmZ6dw0sh8L125l8oeHehVGRKTxUZAk6IKjOjOwc0semraU3Xv2Rl2OiEjKUJAkKC3N\nuHXUANZu3sXf3vko6nJERFKGgqQaTujdjtP6t+fh6UV8vqMs6nJERFKCgqSaxo/qz47Scn732vKo\nSxERSQkKkmrq06EFlw7rxj9nrWbVxh1RlyMiEjkFSQ38+Mw+ZGWk8eBLeklRRERBUgPtW+Tw/RG9\nmLpwPXNWfx51OSIikVKQ1ND/jOhB+xbZ3PNiIY1h4EsRkYNRkNRQ06wMfnJWXz74eDNTF66PuhwR\nkcgoSJJwydFd6dehBQ+8tISy8oqoyxERiYSCJAnpacb4c/qz+rOd/HPW6qjLERGJhIIkSSf3zeXE\n3u343evL2bJrT9TliIjUuVCDxMxGmtlSMysys1sqWX+1mS0ws3lm9paZ5cWtGx9st9TMzk50n3XN\nLHZWsmXXHv4wvSjqckRE6lxoQWJm6cDDwCggD7g8PigCT7r7IHcfAjwI/CrYNg+4DDgSGAn8wczS\nE9xnnTuyUysuGtqFx9/5iDWf74y6HBGROhXmGckxQJG7r3T3MmACcEF8B3ffGrfYDNj3HO0FwAR3\nL3X3VUBRsL8q9xmVG8/uiwG/eHlp1KWIiNSpMIOkM7Ambrk4aPsSM7vWzFYQOyO5voptE9pnFDq2\nasL3TurBC/PWMb94c9TliIjUmTCDxCppO+DNPXd/2N17AeOA26vYNqF9ApjZWDMrMLOCkpKSBEtO\nztUn96Jtsyy9pCgijUqYQVIMdI1b7gIcanrBCcCYKrZNeJ/u/qi757t7fm5ubjVLr5kWOZn86Iw+\nvLfqc14r3FAnvykiErUwg2Q20MfMephZFrGb55PjO5hZn7jFc4F9Y7NPBi4zs2wz6wH0Ad5PZJ9R\nu+yYbvRs14z7phZSvlcvKYpIwxdakLh7OXAdMA0oBJ5290VmdqeZjQ66XWdmi8xsHnADcGWw7SLg\naWAx8BJwrbvvPdg+wzqGmshMT+OWUf1ZUbKDCbPXVL2BiEg9Z43hWn5+fr4XFBTU2e+5O5c+MouV\nG7cz46ZTaZ6dUWe/LSJSW8xsjrvnV9VPb7aHwMy49dwBbNxexiMzV0RdjohIqBQkIRnStTXnH9WJ\nP7+5kvVbdkddjohIaBQkIbr57H5UVMAv9ZKiiDRgCpIQdW3TlCtPOIJnPyim8JOtVW8gIlIPKUhC\ndt2pfWiZk8m9UwqjLkVEJBQKkpC1aprJ/zutN28u38jMZXXzhr2ISF1SkNSBbx5/BF3bNOG+KYXs\nrWj4j1uLSOOiIKkD2RnpjBvZnyXrtzHxg+KoyxERqVUKkjpy7qCODOnaml++vJSdZeVRlyMiUmsU\nJHXEzLjt3AF8urWUx95cFXU5IiK1RkFSh4Z1b8PZR3bgTzNXULKtNOpyRERqhYKkjo0b2Z/S8gp+\n8+qyqEsREakVCpI61jO3Od84thsTZq+haMO2qMsREUmagiQC15/eh6aZ6dw/dUnUpYiIJE1BEoG2\nzbO55tRevFq4gXdXfBZ1OSIiSVGQROS7w3vQqVUO904ppEIvKYpIPaYgiUhOZjo3nt2PBWu38O/5\nh5rKXkQktSlIIjRmSGeO7NSSB19ayu49e6MuR0SkRkINEjMbaWZLzazIzG6pZP0NZrbYzOab2Wtm\ndkTQfqqZzYv77DazMcG6v5nZqrh1Q8I8hjClpRm3nTOAtZt38fd3Poq6HBGRGgktSMwsHXgYGAXk\nAZebWd5+3eYC+e4+GHgWeBDA3ae7+xB3HwKcBuwEXo7b7qZ96919XljHUBdO6N2OU/vl8vvpRWza\nURZ1OSIi1RbmGckxQJG7r3T3MmACcEF8hyAwdgaLs4AuleznEmBqXL8GZ/w5A9hRWs7vXl8edSki\nItUWZpB0BtbELRcHbQdzFTC1kvbLgKf2a7snuBz2azPLTq7M6PXt0IJLh3Xln7NW89HGHVGXIyJS\nLWEGiVXSVulzrmZ2BZAPPLRfe0dgEDAtrnk80B8YBrQBxh1kn2PNrMDMCkpKUn9CqR+f0ZfM9DQe\nnKaXFEWkfgkzSIqBrnHLXYADnnM1szOA24DR7r7/SIZfAya5+559De7+iceUAo8Tu4R2AHd/1N3z\n3T0/Nzc3yUMJX/uWOYwd0ZMpC9YzZ/WmqMsREUlYmEEyG+hjZj3MLIvYJarJ8R3MbCjwCLEQ2VDJ\nPi5nv8tawVkKZmbAGGBhCLVH4n9O6klui2zueXEx7npJUUTqh9CCxN3LgeuIXZYqBJ5290VmdqeZ\njQ66PQQ0B54JHuX9ImjMrDuxM5qZ++36CTNbACwA2gF3h3UMda1ZdgY/ObMvH3y8mZcWro+6HBGR\nhFhj+C/f/Px8LygoiLqMhOytcEb99g1Kyyt45ccnk5Whd0ZFJBpmNsfd86vqp7+lUkx6mjH+nAGs\n/mwnT7y3OupyRESqpCBJQaf0zWV477b87rXlbNm1p+oNREQipCBJQWbG+FED2LxrD3+YURR1OSIi\nh6QgSVEDO7fiwqGdefztjyje1GBf6heRBkBBksJuPKsfBvxi2tKoSxEROSgFSQrr1LoJV53Yg+fn\nrWNB8ZaoyxERqZSCJMVdfUov2jTL4p4peklRRFKTgiTFtczJ5Edn9GHWys95fUllL/+LiERLQVIP\nXH5MN3q2a8a9Uwop31sRdTkiIl+iIKkHMtPTGDeqPytKdvCvgjVVbyAiUocUJPXEWXkdGNb9MH79\nynK2l5ZHXY6IyBcUJPWEmXHrOQPYuL2UR2euiLocEZEvKEjqkaHdDuO8wR159M2VrN+yO+pyREQA\nBUm9c/PZ/dlb4fzqFb2kKCKpQUFSz3Rr25Qrj+/OM3OKKfxka9TliIgoSOqj607rTYvsDO6bqvnd\nRSR6CpJ6qHXTLK4/vQ9vLCvhjWUlUZcjIo1cqEFiZiPNbKmZFZnZLZWsv8HMFpvZfDN7zcyOiFu3\nN5h+d/8peHuY2XtmttzM/hXMB9/ofPP4I+japgn3Tilkb4WGThGR6IQWJGaWDjwMjALygMvNLG+/\nbnOBfHcfDDwLPBi3bpe7Dwk+o+PaHwB+7e59gE3AVWEdQyrLzkjn5rP7s2T9Np77oDjqckSkEQvz\njOQYoMjdV7p7GTABuCC+g7tPd/d9k23MArocaodmZsBpxEIH4O/AmFqtuh45b3BHjuraml++vIxd\nZXujLkdEGqkwg6QzED+eR3HQdjBXAVPjlnPMrMDMZpnZvrBoC2x2932vdle1zwbNzLjtnAGs37qb\nx95aGXU5ItJIhRkkVklbpRfzzewKIB94KK65m7vnA18HfmNmvaq5z7FBEBWUlDTcG9LH9GjDWXkd\n+OOMFZRsK426HBFphMIMkmKga9xyF2Dd/p3M7AzgNmC0u3/xN6G7rwv+uRKYAQwFNgKtzSzjUPsM\ntnvU3fPdPT83Nzf5o0lh40b1Z3d5Bb99bVnUpYhIIxRmkMwG+gRPWWUBlwGT4zuY2VDgEWIhsiGu\n/TAzyw6+twOGA4s9NrPTdOCSoOuVwAshHkO90Cu3Od84thtPvb+Gog3boy5HRBqZ0IIkuI9xHTAN\nKASedvdFZnanme17CushoDnwzH6P+Q4ACszsQ2LBcb+7Lw7WjQNuMLMiYvdMHgvrGOqTH57ehyaZ\n6dyvlxRFpI5ZY5i+NT8/3wsKCqIuI3QPTy/ioWlLmTD2OI7r2TbqckSknjOzOcG96kPSm+0NyFUn\n9qBjqxzunVJIhV5SFJE6oiBpQHIy07nxrH7ML97Cv+dX+gyCiEitU5A0MBcO7Uxex5Y8+NJSdu/R\nS4oiEr5qB4mZpZlZyzCKkeSlpRm3nTuAtZt38Y93P4q6HBFpBBIKEjN70sxamlkzYDGw1MxuCrc0\nqanhvdtxSr9cfv96EZt2lEVdjog0cImekeS5+1Zi41pNAboB3wytKkna+FED2F5azv++XhR1KSLS\nwCUaJJlmlkksSF5w9z0cZGgSSQ39Dm/B1/K78n+zPmL1ZzuiLkdEGrBEg+QR4COgGfBGMG+I5nlN\ncTec2ZeMtDQefEnzu4tIeBIKEnf/nbt3dvdzPGY1cGrItUmS2rfMYeyInry44BPmrN4UdTki0kAl\nerP9h8HNdjOzx8zsA2LzgkiKGzuiJ7ktsrl3SiGNYRQDEal7iV7a+m5ws/0sIBf4DnB/aFVJrWmW\nncENZ/ZlzupNTF24PupyRKQBSjRI9s0Dcg7wuLt/SOVzg0gK+urRXeh/eAt+/u9FbN29J+pyRKSB\nSTRI5pjZy8SCZJqZtQAqwitLalNGehr3XzyYDdtKefAljQ4sIrUr0SC5CrgFGBbMsZ5F7PKW1BND\nurbmOyf04J+zPqbgo8+jLkdEGpBEn9qqIDYb4e1m9gvgBHefH2plUut+clZfOrduwriJ8ykt1zhc\nIlI7En1q637gh8SGR1kMXG9m94VZmNS+ZtkZ3HPhQFaU7ODh6SuiLkdEGohEL22dA5zp7n91978C\nI4FzwytLwnJKv/aMGdKJP84oYtmn26IuR0QagOqM/ts67nur2i5E6s5Pz8ujeXYG4ybOZ68mwBKR\nJCUaJPcBc83sb2b2d2AOcG9VG5nZSDNbamZFZnZLJetvMLPFZjbfzF4Lhl7BzIaY2btmtihYd2nc\nNn8zs1XBHO/zzGxIgscggbbNs/nZ+XnM/Xgz/5y1OupyRKSeS/Rm+1PAccBzwed4d59wqG3MLB14\nGBgF5AGXm1neft3mAvnuPhh4FngwaN8JfMvdjyR2Ge03ZhZ/RnSTuw8JPvMSOQb5sjFDOjOiby4P\nvrSEdZt3RV2OiNRjhwwSM/vKvg/QESgG1gCdgrZDOQYocveV7l4GTAAuiO/g7tODx4kBZhF7Mgx3\nX+buy4Pv64ANxN6ol1piZtwzZiAVDj99fqGGTxGRGsuoYv0vD7HOOfR4W52Jhc4+xcCxh+h/FTB1\n/0YzO4bYeyvxjxndY2Y/A14DbnH30kPsVw6ia5um/OSsvtz9YiH/mf8J5x/VKeqSRKQeOmSQuHsy\nI/xWNoRKpf/Za2ZXAPnAyfu1dwT+D7gyeJcFYDywnli4PAqMA+6sZJ9jgbEA3bp1q9kRNALfGd6D\nyR+u4+f/XsRJfdrRumlW1CWJSD2T6HskF1XyOd3M2h9is2Kga9xyF2BdJfs+A7gNGB1/ZhHMC/8i\ncLu7z9rX7u6fBEPZlwKPE7uEdgB3f9Td8909PzdXV8UOJj3NuP+iwWzauYd7XiyMuhwRqYeqM0TK\nX4BvBJ8/AzcAb5vZwabcnQ30MbMeZpYFXAZMju9gZkOJTZo12t03xLVnAZOAf7j7M/tt0zH4pxGb\nsXFhgscgB5HXqSXfH9GTZ+YU89byjVGXIyL1TKJBUgEMcPeL3f1iYk9hlRK75zGusg3cvRy4DpgG\nFAJPu/siM7vTzEYH3R4CmgPPBI/y7guarwEjgG9X8pjvE2a2AFgAtAPurs4BS+WuP70PPdo149ZJ\nC9hVpuFTRCRxlsjTOma2wN0HxS0bsMDdB5rZXHcfGmaRycrPz/eCgoKoy0h57674jMv/PIvvj+jJ\n+HMGRF2OiETMzOa4e35V/RI9I3nTzP5jZlea2ZXELlG9YWbNgM3JFCqp4/hebblsWFf+8tYqFq7d\nEnU5IlJPJBok1xK7sT0EGAr8HbjW3Xck+WSXpJjxowbQplkW4ybOp3yvppwRkaol+ma7A28BrwOv\nAm+43mBrkFo1zeTO0UeyaN1WHntrVdTliEg9kOjjv18D3gcuIXYj/D0zuyTMwiQ6Iwcezpl5Hfj1\nq8tY/dmOqMsRkRSX6KWt24jNjnilu3+L2LsbPw2vLImSmXHXBQPJTEvj1kkLNHyKiBxSokGSFv+e\nB/BZNbaVeujwVjmMG9Wft4s+49k5xVGXIyIpLNEweMnMppnZt83s28TeOJ8SXlmSCr5+TDeGdT+M\nu18spGSbhjMTkcolerP9JmLjWg0GjgIedfdKX0SUhiMtzbjvokHsKtvLnf9ZHHU5IpKiqhr99wvu\nPhGYGGItkoJ6t2/Bdaf15levLGPMkE6cPqBD1CWJSIqpaj6SbWa2tZLPNjPbWldFSrSuPrkXfTs0\n5/bnF7K9tDzqckQkxRwySNy9hbu3rOTTwt1b1lWREq2sjDTuv3gw67fu5hfTlkZdjoikGD15JQn5\nSrfDuPL47vz93Y+Ys3pT1OWISApRkEjCbjy7Hx1b5jD+ufmUlWv4FBGJUZBIwppnZ3D3hQNZ9ul2\n/jRzRdUbiEijoCCRajmtfwfOP6oTv3+9iKIN26IuR0RSgIJEqu2O8/Nomp3OLRMXUFGh4VNEGjsF\niVRbu+bZ3H5uHgWrN/HE+x9HXY6IRExBIjVy8Vc6c2LvdjwwdQnrt+yOuhwRiVCoQWJmI81sqZkV\nmdktlay/wcwWm9l8M3vNzI6IW3elmS0PPlfGtR9tZguCff4umPZX6piZcc+FAymvqOD25xdqhGCR\nRiy0IDGzdOBhYBSQB1xuZnn7dZsL5Lv7YOBZ4MFg2zbAHcCxxIasv8PMDgu2+SMwFugTfEaGdQxy\naEe0bcYNZ/bl1cJPmbpwfdTliEhEwjwjOQYocveV7l4GTAAuiO/g7tPdfWewOAvoEnw/G3jF3T93\n903AK8BIM+sItHT3d4MZGv8BjAnxGKQK3x3eg4GdW3LH5EVs2bkn6nJEJAJhBklnYE3ccnHQdjBX\nAVOr2LZz8L3KfZrZWDMrMLOCkpKSapYuicpIT+P+iwbz+Y4y7ptaGHU5IhKBMIOksnsXlV5IN7Mr\ngHzgoSq2TXif7v6ou+e7e35ubm4C5UpNDezciu+d1IMJs9fwzoqNUZcjInUszCApBrrGLXcB1u3f\nyczOIDaV72h3L61i22L+e/nroPuUuvej0/tyRNum3PrcAnbv2Rt1OSJSh8IMktlAHzPrYWZZwGXA\n5PgOZjYUeIRYiMRP5TsNOMvMDgtusp8FTHP3T4BtZnZc8LTWt4AXQjwGSVCTrHTuvXAQH322k9+9\ntjzqckSkDoUWJO5eDlxHLBQKgafdfZGZ3Wlmo4NuDwHNgWfMbJ6ZTQ62/Ry4i1gYzQbuDNoArgH+\nAhQBK/jvfRWJ2PDe7fjq0V145I2VLF6n6WpEGgtrDM//5+fne0FBQdRlNAqbd5Zxxq9m0ql1Eyb9\nYDjpaXrNR6S+MrM57p5fVT+92S61qnXTLO44/0jmF2/h8bdXRV2OiNQBBYnUuvMGd+T0/u355cvL\nWPP5zqo3EJF6TUEitc7MuGvMQNIMbp20QMOniDRwChIJRafWTRg3qj9vLt/IpLlroy5HREKkIJHQ\nXHHsEXylW2vu+s9iPtteWvUGIlIvKUgkNGlpxv0XD2Z7aTl3/Wdx1OWISEgUJBKqvh1a8INTevP8\nvHVMX7qh6g1EpN5RkEjofnBqL3q3b87tkxayo7Q86nJEpJYpSCR02Rnp3H/RINZu3sUvX14WdTki\nUssUJFIn8ru34ZvHHcHj76xi7seboi5HRGqRgkTqzM0j+9GhRQ7jn1vAnr0VUZcjIrVEQSJ1pkVO\nJneNGciS9dt49I2VUZcjIrVEQSJ16sy8Dpw7qCO/fW05K0q2R12OiNQCBYnUuTtG55GTkcb45xZQ\nUaHhU0TqOwWJ1Ln2LXK4/dw83l/1Of8qWBN1OSKSJAWJROKr+V04vmdb7p1SyKdbd0ddjogkQUEi\nkTAz7r1oEGXlFdzxwqKoyxGRJIQaJGY20syWmlmRmd1SyfoRZvaBmZWb2SVx7acGU+/u++w2szHB\nur+Z2aq4dUPCPAYJT492zfjRGX15adF6Xlq4PupyRKSGQgsSM0sHHgZGAXnA5WaWt1+3j4FvA0/G\nN7r7dHcf4u5DgNOAncDLcV1u2rfe3eeFdQwSvu+d1IMBHVvysxcWsmXXnqjLEZEaCPOM5BigyN1X\nunsZMAG4IL6Du3/k7vOBQ72ddgkw1d011V4DlJmexgMXD2Lj9lIeeGlJ1OWISA2EGSSdgfhHcoqD\ntuq6DHhqv7Z7zGy+mf3azLJrWqCkhsFdWnPViT148r2PeW/lZ1GXIyLVFGaQWCVt1XppwMw6AoOA\naXHN44H+wDCgDTDuINuONbMCMysoKSmpzs9KBH58Zl+6tmnC+EkL2L1nb9TliEg1hBkkxUDXuOUu\nwLpq7uNrwCR3/+Liubt/4jGlwOPELqEdwN0fdfd8d8/Pzc2t5s9KXWualcG9Fw5iZckOHp5eFHU5\nIlINYQbJbKCPmfUwsyxil6gmV3Mfl7PfZa3gLAUzM2AMsLAWapUUcFKfXC76Smf+OGMFS9Zvjboc\nEUlQaEHi7uXAdcQuSxUCT7v7IjO708xGA5jZMDMrBr4KPGJmX7xQYGbdiZ3RzNxv10+Y2QJgAdAO\nuDusY5C699Nz82jVJJNbJi5gr4ZPEakXzL3h/2HNz8/3goKCqMuQBL0wby0/nDCPO87P4zvDe0Rd\njkijZWZz3D2/qn56s11SzuijOnFKv1wemraU4k166lsk1SlIJOWYGXePGQjA7c8vpDGcNYvUZwoS\nSUldDmvKTWf3Y8bSEiZ/WN2H/USkLilIJGV96/juDOnamp//ezGf7yiLuhwROQgFiaSs9DTj/osH\nsXXXHu5+cXHU5YjIQShIJKX1P7wl15zSi+c+WMubyzVCgUgqUpBIyrv21N70zG3GrZMWsLOsPOpy\nRGQ/ChJJeTmZ6dx34SDWfL6LX7+yLOpyRGQ/ChKpF47t2ZavH9uNx95axfzizVGXIyJxFCRSb9wy\nqj/tmmczbuICtpfqEpdIqlCQSL3RMieTu8cMpPCTrZxw32v8YtpSPtteGnVZIo2egkTqlbOOPJwX\nrh3O8N7teHhGEcMfeJ07XljIms81lIpIVDRoo9RbK0q288jMFUyau5YKj43R9f2Te9L/8JZRlybS\nICQ6aKOCROq9T7bs4rE3V/Hk+x+zs2wvp/dvzzWn9CK/e5uoSxOp1xQkcRQkjcPmnWX8493VPP72\nKjbt3MOw7odxzSm9OLVfe2LzoIlIdShI4ihIGpedZeU8PXsNf35zFWs376L/4S24+uRenDe4Ixnp\nui0okigFSRwFSeO0Z28F//5wHX+auYJln26ny2FNGDuiJ189uitNstKjLk8k5SlI4ihIGreKCuf1\nJRv4w4wiPvh4M22bZfGd4d355nHdadU0M+ryRFJWSsyQaGYjzWypmRWZ2S2VrB9hZh+YWbmZXbLf\nur1mNi/4TI5r72Fm75nZcjP7l5llhXkMUv+lpRln5HVg4jUn8PT3j2dwl1b84uVlDH/gde6bUsin\nW3dHXaJIvRbaGYmZpQPLgDOBYmA2cLm7L47r0x1oCdwITHb3Z+PWbXf35pXs92ngOXefYGZ/Aj50\n9z8eqhadkcj+Fq/byiNvrODfH64jIy2Ni4/uzNgRvejRrlnUpYmkjFQ4IzkGKHL3le5eBkwALojv\n4O4fuft8oCKRHVrs0ZvTgH2B83dgTO2VLI1FXqeW/Payocy48VQuHdaViR+s5bRfzuDaJz5gQfGW\nqMsTqVfCDJLOwJq45eKgLVE5ZlZgZrPMbF9YtAU2u/u+gZYOuk8zGxtsX1BSonkspHLd2jblrjED\neXvcaVxzci/eWFbC+b9/i28+9h7vFG3UfPEiCQgzSCp7cL86fyq7BadUXwd+Y2a9qrNPd3/U3fPd\nPT83N7caPyuNUW6LbG4e2Z+3x5/GLaP6s2T9Nr7+l/cY8/DbvLTwEyoqFCgiBxNmkBQDXeOWuwDr\nEt3Y3dcF/1wJzACGAhuB1maWUZN9ilSlZU4mV5/cizdvPpV7LxzE5l17uPqfH3DGr2fy9Ow1lJUn\ndBVWpFEJM0hmA32Cp6yygMvoabsQAAAJCElEQVSAyVVsA4CZHWZm2cH3dsBwYLHHrjNMB/Y94XUl\n8EKtVy6NXk5mOl8/thuv/+QUfv/1oTTJTOfmifMZ8eB0/vLmSnZoGHuRL4T6HomZnQP8BkgH/uru\n95jZnUCBu082s2HAJOAwYDew3t2PNLMTgEeI3YRPA37j7o8F++xJ7MZ9G2AucIW7H3IscT21Jcly\nd95cvpE/zljBuys/o1WTTK48oTvfPqE7bZrpCXRpmPRCYhwFidSmuR9v4k8zVzBt0afkZKZx2bBu\nfO+kHnQ5rGnUpYnUKgVJHAWJhKFowzYembmSSXPXAjB6SCeuPrkXfTu0iLgykdqhIImjIJEwrdu8\ni8feWsVTwTD2ZwzowDWn9OLoIw6LujSRpChI4ihIpC5s2hEbxv5v78SGsT+mRxuuOaUXp/TN1TD2\nUi8pSOIoSKQu7SwrZ8L7a/jLmytZt2U3/Q9vwTWn9OLcQRrGXuoXBUkcBYlEoay8gsnBMPZFG7bT\ntU0Txo7oxVeP7kJOpoaxl9SnIImjIJEoVVQ4rxZ+yh9mrGDems20a57Fd4b34IrjjqBVEw1jL6lL\nQRJHQSKpwN15b9Xn/HHGCmYuK6F5dgbnDe5Iu+bZNMvOoHlOBi2yM2gefG+enUGLnP8uZ2foLEbq\nVqJBklFVBxGpHWbGcT3bclzPtixat4U/zVzJ1IXr2V5azt4ExvLKTLe4kMmMhU5c0LTIzogFUnwo\nfSmQMmmWnU6zrAzS0nTzX2qPgkQkAkd2asX/Xj4UiJ2p7N5TwbbSPWzfXc6O0r1ffN9eGvts2/c9\nrm377nJKtpWyauOOYP0edu9JbCyw5gc78wnCKP5MKD6I9t8uK0MPD4iCRCRyZkaTrHSaZKXTPsl3\nGffsrWBHEDw7ymJhsy0+gL60vCcIpb1s372HT7fu/u/60nISueqdlZH2xZlPZvBEWvzlcj/gy5eH\n697X98tt8X39wLZK6qr0NxPZV6V9v/wD+/9eZfVXvuX+dR+678H+N6pq2wPr+3LDi9efRK/cA+YI\nrFUKEpEGJDM9jdZNs2jdNLnxv9ydnWV7vzgb2nHAmVEshOJDqnxv3F9gduDX+Hdp4i+s7Wv+clsl\nfb+0z7j1lW5/YN8vtX3pyt6h93Vg/8p/v5Iyg/UHv4yY3H4P/hvxS3XxQIeCREQOYGY0Cy5zdWgZ\ndTWS6nSBU0REkqIgERGRpChIREQkKQoSERFJioJERESSoiAREZGkKEhERCQpChIREUlKoxj918xK\ngNU13LwdsLEWy4lSQzmWhnIcoGNJVQ3lWJI9jiPcPbeqTo0iSJJhZgWJDKNcHzSUY2koxwE6llTV\nUI6lro5Dl7ZERCQpChIREUmKgqRqj0ZdQC1qKMfSUI4DdCypqqEcS50ch+6RiIhIUnRGIiIiSVGQ\nJMDM7jKz+WY2z8xeNrNOUddUU2b2kJktCY5nkpm1jrqmmjCzr5rZIjOrMLN6+XSNmY00s6VmVmRm\nt0RdT02Z2V/NbIOZLYy6lmSYWVczm25mhcG/Wz+MuqaaMrMcM3vfzD4MjuXnof6eLm1VzcxauvvW\n4Pv1QJ67Xx1xWTViZmcBr7t7uZk9AODu4yIuq9rMbABQATwC3OjuBRGXVC1mlg4sA84EioHZwOXu\nvjjSwmrAzEYA24F/uPvAqOupKTPrCHR09w/MrAUwBxhTT/8/MaCZu283s0zgLeCH7j4rjN/TGUkC\n9oVIoBmVTLdcX7j7y+5eHizOArpEWU9NuXuhuy+Nuo4kHAMUuftKdy8DJgAXRFxTjbj7G8DnUdeR\nLHf/xN0/CL5vAwqBztFWVTMesz1YzAw+of29pSBJkJndY2ZrgG8AP4u6nlryXWBq1EU0Up2BNXHL\nxdTTv7QaIjPrDgwF3ou2kpozs3QzmwdsAF5x99CORUESMLNXzWxhJZ8LANz9NnfvCjwBXBdttYdW\n1bEEfW4DyokdT0pK5DjqMaukrd6e6TYkZtYcmAj8aL+rEfWKu+919yHErjocY2ahXXbMCGvH9Y27\nn5Fg1yeBF4E7QiwnKVUdi5ldCZwHnO4pfJOsGv+f1EfFQNe45S7AuohqkUBwP2Ei8IS7Pxd1PbXB\n3Teb2QxgJBDKAxE6I0mAmfWJWxwNLImqlmSZ2UhgHDDa3XdGXU8jNhvoY2Y9zCwLuAyYHHFNjVpw\ng/oxoNDdfxV1Pckws9x9T2SaWRPgDEL8e0tPbSXAzCYC/Yg9JbQauNrd10ZbVc2YWRGQDXwWNM2q\nj0+gmdmFwP8CucBmYJ67nx1tVdVjZucAvwHSgb+6+z0Rl1QjZvYUcAqxkWY/Be5w98ciLaoGzOxE\n4E1gAbE/6wC3uvuU6KqqGTMbDPyd2L9bacDT7n5naL+nIBERkWTo0paIiCRFQSIiIklRkIiISFIU\nJCIikhQFiYiIJEVBIlILzGx71b0Ouf2zZtYz+N7czB4xsxXByK1vmNmxZpYVfNeLxJJSFCQiETOz\nI4F0d18ZNP2F2CCIfdz9SODbQLtgcMfXgEsjKVTkIBQkIrXIYh4KxgRbYGaXBu1pZvaH4AzjP2Y2\nxcwuCTb7BvBC0K8XcCxwu7tXAAQjBL8Y9H0+6C+SMnSKLFK7LgKGAEcRe9N7tpm9AQwHugODgPbE\nhij/a7DNcOCp4PuRxN7S33uQ/S8EhoVSuUgN6YxEpHadCDwVjLz6KTCT2F/8JwLPuHuFu68Hpsdt\n0xEoSWTnQcCUBRMviaQEBYlI7apsePhDtQPsAnKC74uAo8zsUH82s4HdNahNJBQKEpHa9QZwaTCp\nUC4wAnif2FSnFwf3SjoQG+Rwn0KgN4C7rwAKgJ8Ho9FiZn32zcFiZm2BEnffU1cHJFIVBYlI7ZoE\nzAc+BF4Hbg4uZU0kNgfJQmLzzL8HbAm2eZEvB8v3gMOBIjNbAPyZ/85VcipQ70ajlYZNo/+K1BEz\na+7u24OziveB4e6+PpgvYnqwfLCb7Pv28Rwwvp7PVy8NjJ7aEqk7/wkmG8oC7grOVHD3XWZ2B7E5\n2z8+2MbBBFjPK0Qk1eiMREREkqJ7JCIikhQFiYiIJEVBIiIiSVGQiIhIUhQkIiKSFAWJiIgk5f8D\nKvAxJUYAaMAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107b65b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# scores_：dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold,\n",
    "# Each dict value has shape (n_folds, len(Cs))\n",
    "Cs = [1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]\n",
    "n_Cs = len(Cs)\n",
    "n_classes = 9\n",
    "scores =  np.zeros((n_classes,n_Cs))\n",
    "\n",
    "for j in range(n_classes):\n",
    "        scores[j][:] = np.mean(lrcv_L1.scores_['Class_'+ str(j+1)],axis = 0)\n",
    "    \n",
    "mse_mean = -np.mean(scores, axis = 0)\n",
    "plt.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('logloss')\n",
    "plt.show()\n",
    "\n",
    "#print ('C is:',lr_cv.C_)  #对多类分类问题，每个类别的分类器有一个C\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.32140535,  0.23499902,  0.16034243,  0.13795419,  0.13536477,\n",
       "        0.13543209,  0.13548341])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mse_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个score似乎和GridSearchCV得到的Score不一样:("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 0.135364772754\n"
     ]
    }
   ],
   "source": [
    "best_C = np.argmin(mse_mean)\n",
    "best_score = np.min(mse_mean)\n",
    "print Cs[best_C], best_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -2.51316065e+00,  -1.29199566e+01,  -9.37230688e+00,\n",
       "         -5.40603619e+00,   2.56371207e+00,   1.65336573e+00,\n",
       "          2.26690516e+00,   2.42643066e+00,  -1.33485147e+01,\n",
       "         -8.08253281e-01,  -1.71897672e+01,  -1.50185630e+00,\n",
       "         -1.22908346e+01,  -1.12655999e+01,  -2.09018723e+01,\n",
       "         -2.77514248e+00,   7.50368922e+00,   1.06186144e+00,\n",
       "         -2.30960719e+01,   5.48268436e-01,  -1.23749439e+00,\n",
       "         -3.31197070e-01,   8.76871146e+00,   2.27109069e+01,\n",
       "         -1.13262520e+01,  -3.59171148e+00,  -6.42512388e+00,\n",
       "         -2.37279970e-01,  -8.69053041e+00,   1.46010503e+01,\n",
       "         -3.83070644e+01,   1.18183014e+00,  -6.67974464e+00,\n",
       "         -2.66144404e+01,  -2.18197998e+01,  -5.83634483e+00,\n",
       "          3.69565572e+00,  -4.22994154e+00,  -6.01830986e+00,\n",
       "         -1.66610540e+01,  -1.21963080e+01,  -1.23598061e+01,\n",
       "         -4.48857435e+01,   2.44238339e+00,  -1.24130235e+01,\n",
       "         -5.53599959e+00,  -1.03947386e+01,  -2.32707038e-01,\n",
       "         -2.39146804e+00,  -2.87451792e+01,   7.82591284e-01,\n",
       "         -5.21036335e+00,   4.86621353e+00,  -1.93300211e+00,\n",
       "          1.87708471e+00,  -6.79938771e+00,  -7.54110328e+00,\n",
       "         -6.73602168e+01,  -1.02434478e+01,  -7.51624329e+00,\n",
       "          4.32421286e+00,   4.84474845e-01,  -3.79643997e+00,\n",
       "         -7.60046049e+00,  -7.89458116e-01,   4.05704133e+00,\n",
       "          7.85045786e+00,  -2.52821534e+01,  -1.96398625e+01,\n",
       "          5.13900261e+00,   1.09141487e+00,  -6.24474384e+00,\n",
       "         -1.54734960e+00,  -4.90985583e+01,  -1.51829276e+01,\n",
       "          7.58310400e+00,   6.66319299e+00,  -1.16208160e+00,\n",
       "          1.41756649e+00,  -2.79469888e+00,   6.72167608e+00,\n",
       "         -7.76875378e+00,  -2.70273295e+00,   8.15577944e+00,\n",
       "          1.30603667e+00,  -1.06884093e+01,  -6.43805385e+00,\n",
       "         -1.76958546e+00,  -5.75999089e+00,  -3.26687767e+01,\n",
       "         -3.95996354e+00,   6.02441504e-01,  -1.80897997e+01,\n",
       "         -1.50664293e+01,  -1.63791856e-01,  -8.43626438e+00],\n",
       "       [ -4.38553519e-01,  -1.18467541e+01,  -6.10667419e+00,\n",
       "          4.47042895e-01,   2.43543763e+00,   6.94567637e-01,\n",
       "         -6.00452356e+00,  -8.48045280e+00,   2.43913227e+00,\n",
       "          5.28757750e-01,  -1.07255963e+01,   5.21412335e+00,\n",
       "         -1.91515443e+01,   2.11057398e+00,   3.13749138e+00,\n",
       "          1.20585818e+00,  -1.71074809e+01,  -2.32099728e-01,\n",
       "         -2.10975384e+01,  -4.74583590e+00,   3.76419653e-02,\n",
       "         -1.24117005e+00,  -2.41905032e+01,   1.19695479e+01,\n",
       "          2.66990952e+00,  -2.39477869e+01,  -2.84826338e+01,\n",
       "         -3.98057276e+00,  -3.13656559e+01,  -3.93796325e+01,\n",
       "         -5.20231056e+00,  -1.29196428e+01,   3.47730537e+00,\n",
       "         -1.28073014e+01,  -8.84667753e+00,  -2.81780796e+01,\n",
       "         -1.56947790e+00,   1.40938112e+00,  -1.15263146e+01,\n",
       "          2.36438355e+00,  -1.70009070e+01,  -1.74969293e+01,\n",
       "         -4.69490766e-01,  -1.48349416e+00,  -1.87244736e+00,\n",
       "         -7.51965784e+00,  -1.67706834e+01,   6.99879436e+00,\n",
       "         -2.49450210e+00,  -9.40236708e+00,  -2.01621556e+01,\n",
       "         -6.15630854e-01,  -1.25457533e+01,   2.51298207e+00,\n",
       "         -3.77918524e+00,  -6.12833150e+00,  -1.71474479e+01,\n",
       "         -1.48022898e+01,  -2.82198217e+01,  -1.20270982e+01,\n",
       "         -9.54649820e+00,  -1.36799518e+00,   6.57305219e+00,\n",
       "          4.02728141e+00,   2.69885816e+00,   1.51754014e+00,\n",
       "          5.33132905e+00,  -5.49368825e+01,  -6.59791742e+00,\n",
       "         -7.86986034e-01,  -1.18446484e+01,   2.04025081e+00,\n",
       "         -5.03389632e+00,   1.21418915e+01,  -2.85741755e+01,\n",
       "         -9.35416058e+00,   8.66798632e+00,  -2.87629904e+01,\n",
       "         -4.97084468e+00,  -7.96750460e+00,  -6.96113035e+00,\n",
       "         -2.43941624e-01,  -3.46185429e+00,  -6.44297199e+01,\n",
       "          3.73575514e-01,   8.08589175e-01,   9.02848793e-01,\n",
       "         -7.46468503e-01,   1.28911693e+00,  -3.51542222e+01,\n",
       "         -2.24444777e+01,   1.36563737e+00,  -1.90574374e+00,\n",
       "         -1.49460388e+01,  -9.19277491e+00,  -5.30121231e+00],\n",
       "       [ -5.32521168e+00,  -3.32433629e+00,  -5.06745401e+00,\n",
       "         -4.53113843e+00,  -4.38382495e+00,  -7.82279239e-01,\n",
       "         -7.66011174e+00,  -5.48963985e+00,   3.66719337e+00,\n",
       "         -1.70842083e+00,  -8.62057122e+00,  -3.06558943e+00,\n",
       "          7.73154357e-01,   2.24067953e+00,   2.37402527e+00,\n",
       "          2.19045846e+00,  -1.25205375e+01,  -2.25745049e+00,\n",
       "         -1.23214391e+00,  -4.28555775e+00,   5.46076219e-01,\n",
       "         -1.57508333e+00,   2.87479201e-02,   1.20641268e+01,\n",
       "         -4.09847552e-01,  -1.78872225e+01,   6.24037701e+00,\n",
       "         -3.36207630e+00,   3.67365227e+00,   1.26865209e+01,\n",
       "          1.98353436e+00,   1.39212024e+01,  -2.03492070e+00,\n",
       "         -8.16465008e+00,  -7.86901111e+00,  -2.30156767e+00,\n",
       "         -8.62971760e-01,   1.73797177e+00,  -6.05049127e+00,\n",
       "          3.68198987e+00,   1.57618720e-01,  -1.04974574e+01,\n",
       "          7.90603884e+00,  -7.52210199e-01,  -1.02548455e+01,\n",
       "         -1.51043632e-01,  -2.04887654e+01,  -1.71533733e+00,\n",
       "          0.00000000e+00,   4.98700164e-02,  -1.24041118e+01,\n",
       "         -6.48933685e-01,  -2.55488208e+00,  -8.53282201e-01,\n",
       "         -9.60962287e-01,   4.25376483e+00,  -1.43354891e+01,\n",
       "         -1.03342022e+01,  -1.71426325e+01,  -3.78274812e+01,\n",
       "         -7.77770239e+00,   8.68103964e-01,   1.43095144e+00,\n",
       "          5.51954705e+00,  -2.82151691e+00,  -1.18427610e+00,\n",
       "         -8.68410497e+00,  -9.50410677e+00,  -2.04022330e+01,\n",
       "          1.24199735e+00,   3.08968845e+00,  -4.02490353e-01,\n",
       "         -6.38876602e+01,  -3.85467362e+01,  -1.72859275e+01,\n",
       "         -1.58979825e+01,  -1.64462767e+01,  -3.21240019e+00,\n",
       "         -1.31835232e+00,   3.01794626e+00,  -2.94247486e+00,\n",
       "         -8.80985962e+00,   1.24622697e+01,  -5.88798508e+01,\n",
       "          4.75952795e+00,   5.31082289e+00,  -7.97313396e+00,\n",
       "          1.55619347e+00,   7.90042107e+00,  -2.23227127e+01,\n",
       "         -1.38493960e+01,  -1.41481464e+01,  -2.30590316e+01,\n",
       "         -1.99920094e+01,  -3.96713275e-01,  -2.60505573e+00],\n",
       "       [  9.38284159e+00,  -1.61321397e+01,  -1.60529594e+01,\n",
       "         -1.75539084e+01,   2.57670532e+00,   2.17063499e-01,\n",
       "         -6.53254788e+00,  -3.29911364e+00,  -8.00244260e+00,\n",
       "          1.97526299e+00,  -1.61963262e+01,  -8.81752041e+00,\n",
       "          1.99681265e+00,   1.29779362e+00,   6.78448841e+00,\n",
       "         -9.87522736e-01,  -1.45601695e+01,  -9.19440812e+00,\n",
       "          1.66961448e+00,   8.74377288e-01,  -2.24798182e+00,\n",
       "          6.70429564e-01,   1.84169874e+00,  -4.33397499e+01,\n",
       "          4.67251950e+00,   1.41364487e+01,   1.05348022e+00,\n",
       "         -7.95843281e+00,  -1.32002156e+01,  -2.28595749e+01,\n",
       "         -1.01392260e+01,  -1.04129857e+01,  -1.60584749e+00,\n",
       "         -7.16679989e+00,  -3.17084526e+01,  -1.93749652e-01,\n",
       "         -1.00494703e+00,  -2.38997378e+00,  -2.18574458e+01,\n",
       "          1.79393192e+00,  -2.82689542e+00,   3.88484242e+00,\n",
       "          1.30747555e+00,   1.89870277e+00,  -6.41572311e+00,\n",
       "         -1.91066202e+00,  -3.42339916e+01,  -8.69278964e+00,\n",
       "         -2.72429982e+01,  -6.22908007e+00,  -3.47989033e+01,\n",
       "          9.93695812e+00,  -1.94216821e+00,  -2.78803834e+00,\n",
       "         -1.37212251e+00,   5.18963578e+00,  -1.86137638e+01,\n",
       "         -6.09212510e+01,  -3.24934045e+01,  -8.47295316e-01,\n",
       "         -2.35258409e+01,  -1.80715736e+01,   5.34375713e+00,\n",
       "         -5.83958665e-01,  -7.71163730e+00,  -2.74037402e+00,\n",
       "         -1.93920138e+01,  -2.35623920e+00,   2.03290383e+00,\n",
       "         -1.70374673e-01,  -8.72264781e+00,  -3.48369312e-01,\n",
       "          1.53392201e+01,   3.25452887e+00,  -6.76543750e+00,\n",
       "         -3.90398959e+01,  -1.67267068e+01,  -6.35812159e+00,\n",
       "          2.13537467e+00,  -2.35751826e+00,  -1.05517159e+01,\n",
       "         -1.10253648e+01,  -8.63530721e+00,  -6.09509759e+01,\n",
       "          5.92526634e+00,   5.57006920e+00,   1.25700777e+01,\n",
       "          2.31066469e+00,  -9.78136777e+00,  -3.05428295e+01,\n",
       "          2.59809276e-01,   2.40418371e+00,  -3.78995874e+01,\n",
       "         -1.77937446e+01,  -1.21641805e+01,  -6.45052534e+00],\n",
       "       [ -1.28043892e+01,   3.43738636e+01,   4.46577853e+01,\n",
       "         -2.47071118e+00,  -4.28406907e+00,   3.92001942e-01,\n",
       "         -3.13668186e+01,   2.23107106e+01,   2.46594869e+01,\n",
       "         -1.73643073e+01,  -5.35767558e+01,   8.48577317e+00,\n",
       "          4.74402078e+01,  -1.13916099e+01,  -4.21306628e+00,\n",
       "          3.91534550e+00,  -6.10698077e+01,   1.10252467e+01,\n",
       "          3.49863280e+01,  -2.56901664e-02,  -1.19311449e+01,\n",
       "          7.67116365e+00,   6.82964888e+00,  -5.32276768e+00,\n",
       "         -1.54078094e+01,  -3.31638418e+01,  -3.02599423e+01,\n",
       "         -1.02055278e+01,  -3.70892283e+01,  -1.45057560e+02,\n",
       "          4.03885133e+01,   2.19726424e+01,   6.52345066e+00,\n",
       "          1.28931605e+02,  -2.12576300e+01,  -2.17884398e+01,\n",
       "         -1.16660731e+00,  -5.62359300e+00,  -2.16934304e+02,\n",
       "          1.46473037e+01,  -2.43431740e-01,  -1.31484341e+01,\n",
       "         -1.55764974e+02,  -7.11966899e+00,  -3.28253863e+01,\n",
       "         -3.68058982e+00,  -6.09715119e+01,  -2.40498676e+00,\n",
       "         -9.22898223e+00,  -5.73667309e+01,  -2.95068164e+01,\n",
       "         -7.51463451e+01,  -3.95065685e+00,  -1.30378582e+01,\n",
       "         -1.75900743e+01,  -3.35879321e+01,   1.81565222e+00,\n",
       "         -8.42439590e+01,  -1.66761764e+02,  -4.51106716e+01,\n",
       "         -1.99029765e+01,  -9.66328857e+00,  -3.35373898e+00,\n",
       "         -3.45794978e-01,   6.69490724e-01,   5.88647901e+00,\n",
       "          5.24076634e+00,  -1.85460294e+01,  -1.72224484e+01,\n",
       "         -1.72400558e+00,   9.89670471e+00,  -1.73947226e+01,\n",
       "         -3.13351950e+02,  -2.50534463e+01,  -1.08429009e+02,\n",
       "         -5.64734212e+01,  -2.39825379e+01,  -3.09752941e+01,\n",
       "          7.80526989e+00,   1.24300464e-01,  -2.32159016e+01,\n",
       "         -2.19923040e+01,  -1.62749046e+02,  -2.52726526e+02,\n",
       "          4.18981263e+00,  -5.58860751e+01,   8.48751742e+00,\n",
       "         -1.16316595e+01,  -2.09442363e+01,  -3.08877582e+01,\n",
       "         -3.28202995e+01,   4.42660423e+00,   4.01492237e+01,\n",
       "         -2.98908553e+02,  -4.16347557e+00,  -2.31862703e+00],\n",
       "       [  3.77999500e+00,   6.42746655e-01,   6.24750210e+00,\n",
       "          9.96776184e-01,  -6.33509538e+00,   3.13560000e-01,\n",
       "         -9.90193072e+00,  -5.81522562e+00,  -6.58029830e+00,\n",
       "         -3.15394424e+00,   2.82458713e+01,  -4.51222725e+00,\n",
       "          4.56342245e-01,  -1.03984164e+01,  -1.27711072e+01,\n",
       "         -9.94942106e-01,  -4.54084256e-01,   1.99980526e+00,\n",
       "         -1.81523731e+01,  -2.04555521e+00,  -6.71241593e-01,\n",
       "          1.05207988e-01,   2.96489779e+00,   3.20810035e+00,\n",
       "         -5.81651961e+00,   9.69591651e+00,   7.30103494e+00,\n",
       "         -1.22418958e+00,   5.80224447e+00,  -1.70984221e+01,\n",
       "          2.82570786e+00,  -1.68983168e+01,  -3.91659160e+00,\n",
       "         -2.29017033e+01,  -2.17529890e+01,  -6.95800303e+00,\n",
       "         -4.47638800e+00,  -6.64186077e+00,   9.86208763e-01,\n",
       "         -1.25740722e+01,  -6.28969636e+00,   1.90632725e+01,\n",
       "         -5.87482928e+00,   2.24456508e+00,   3.95246786e+00,\n",
       "          3.22907186e+00,   1.39476996e+00,  -5.72239669e+00,\n",
       "         -2.23329188e-01,   2.80871792e-01,  -1.98361428e+00,\n",
       "         -7.87291725e+00,  -1.50650393e+00,   1.90582631e+00,\n",
       "          2.27888777e-01,  -6.50626392e+00,   1.47921526e+01,\n",
       "         -1.34313159e+01,   2.96463343e+00,   1.50539472e+01,\n",
       "          9.42969454e+00,  -1.26166490e+01,  -3.23661639e-02,\n",
       "          6.64592950e+00,  -2.41846181e+00,  -1.32826084e+00,\n",
       "          4.28180787e-01,  -1.70670347e+01,  -9.75451478e+00,\n",
       "         -2.39603685e+00,  -6.16607922e+00,  -8.67426305e+00,\n",
       "         -1.88068343e+01,   2.08918598e+01,  -5.05336723e+00,\n",
       "         -9.69385727e+00,  -5.57629867e+00,  -3.18365733e+01,\n",
       "          1.78223686e+00,  -1.81005375e+00,  -2.66531322e+00,\n",
       "          1.43595177e+00,  -2.17341966e+00,  -1.64373327e+01,\n",
       "         -3.68115164e+00,  -2.86060156e+00,   6.12916819e+00,\n",
       "         -6.45178634e+00,  -9.09263528e+00,  -2.29154476e+01,\n",
       "         -5.43161194e-01,   7.98172309e-01,   2.99401063e+01,\n",
       "         -1.15521600e+01,  -4.03115413e+00,  -7.12067516e+00],\n",
       "       [ -7.52942617e-01,   1.02382427e+01,  -2.83261012e+00,\n",
       "         -1.98635365e+00,  -6.56988852e+00,  -6.59031604e-01,\n",
       "         -1.43180472e+00,  -1.49872870e+01,  -1.09315336e+01,\n",
       "          2.31792297e+00,  -5.26233327e+00,   2.90571409e+00,\n",
       "         -1.25818271e+01,  -3.95327487e+00,  -1.39808265e+01,\n",
       "          2.33903201e+00,   4.23544189e+00,  -1.92821454e+00,\n",
       "         -5.46738863e+00,  -2.16773520e+00,  -2.71648812e-01,\n",
       "         -2.09075511e+00,   1.90006274e+01,   2.50619928e+01,\n",
       "         -5.28073753e+00,  -7.35511972e+00,  -3.83142656e+01,\n",
       "         -1.38647007e+00,  -4.67279229e+00,  -1.32100534e+01,\n",
       "         -7.15545232e+00,   3.16809815e+01,  -3.08126771e+00,\n",
       "         -1.99408541e+01,  -3.06174884e+00,  -2.09037101e+01,\n",
       "          1.55440419e+00,   4.01606705e+00,   1.88920083e+01,\n",
       "         -1.02359257e+01,  -7.02715806e+00,  -7.38566312e+00,\n",
       "          4.80950852e-01,   8.11917983e-01,   1.17516733e+01,\n",
       "         -6.50708968e+00,   1.97041690e+00,   1.75380768e+00,\n",
       "          1.06793735e+01,   2.01030332e+01,   2.08331602e+00,\n",
       "          3.23026037e+00,  -6.05549023e+00,  -1.00883717e+00,\n",
       "         -4.22149277e-01,   1.62627468e+01,  -2.91971202e+00,\n",
       "         -1.44742770e+01,  -1.73425384e+00,  -3.65263470e+00,\n",
       "         -1.72514970e+01,  -1.30104969e+01,   2.67675615e+00,\n",
       "          5.54996165e+00,  -2.27839776e-01,   1.53485521e+00,\n",
       "          3.30148198e-01,  -1.56049048e+01,  -9.33259970e+00,\n",
       "          2.50561614e+00,   6.69379841e-01,  -3.95820556e+00,\n",
       "         -3.38051083e+01,  -6.52811335e+00,  -1.42144027e+01,\n",
       "         -7.35553442e+00,  -4.11141660e+00,  -4.59958386e+00,\n",
       "          6.05921519e-01,  -2.50332661e+00,   3.76516494e+00,\n",
       "         -1.76178141e+00,   8.21246174e+00,  -6.75816302e+01,\n",
       "          3.20884636e-01,   9.89899943e+00,  -3.94040656e-01,\n",
       "         -7.81452563e-01,  -4.19016283e+00,  -1.58071399e+01,\n",
       "          2.18475119e+00,  -4.70592035e+00,  -5.61560826e+00,\n",
       "         -5.80493159e+01,  -3.55420121e+00,  -5.55024847e+00],\n",
       "       [  3.32511464e+00,  -8.45133361e+00,  -7.60135563e+00,\n",
       "          3.20439663e+00,  -1.40296619e+00,  -3.80813262e-01,\n",
       "          3.42457550e+00,  -6.12158590e+00,  -1.64595028e+00,\n",
       "          2.44156410e+00,  -2.11701868e+01,   3.30002756e-01,\n",
       "          6.37678298e+00,  -1.04951527e+01,  -2.05488253e+01,\n",
       "         -2.42390374e+00,  -6.19455002e+00,  -1.33490608e+00,\n",
       "          2.40957272e+01,   4.65524269e+00,  -2.32196677e-01,\n",
       "         -2.21271630e-01,  -8.11077572e+00,   4.29003455e+00,\n",
       "         -2.70104595e+00,  -9.17226365e+00,  -5.56979904e+00,\n",
       "          4.35219745e+00,  -3.21735955e+00,  -9.53040587e+00,\n",
       "          8.33728682e+00,  -1.93401500e+01,   3.57713784e-01,\n",
       "         -2.31865733e+01,   3.22588519e+01,  -1.70745451e+01,\n",
       "          6.80484181e+00,   7.50170543e+00,  -2.19661297e+01,\n",
       "         -1.22367813e+01,   1.37305067e+00,  -1.20802193e+01,\n",
       "         -2.46771969e+01,  -1.03334439e+00,  -2.22249591e+01,\n",
       "          2.44456173e+00,   8.46040528e+00,  -8.91141208e+00,\n",
       "         -4.81137337e+00,  -1.16101504e+01,   3.81076654e+00,\n",
       "          1.70979519e+00,  -1.26123247e+01,   2.00583220e+00,\n",
       "         -2.38221507e+00,  -5.64216936e+00,  -7.31819946e+00,\n",
       "          2.30770457e+01,  -2.29270826e+01,  -1.35431223e+01,\n",
       "         -6.78730532e+00,  -2.54125875e+00,  -4.13713470e+00,\n",
       "          1.35956716e+00,   4.33803167e+00,   1.71489554e-01,\n",
       "          2.71732533e+00,   4.46069575e+01,   2.46165052e+01,\n",
       "         -4.20128576e-01,   7.81501006e+00,  -6.63604335e+00,\n",
       "          5.18103423e+01,   8.36648093e+00,   2.82973785e+01,\n",
       "          3.30091521e+00,  -1.31783231e+01,   1.25479727e+01,\n",
       "         -5.82375259e+00,   2.33540404e+00,   2.21167239e+00,\n",
       "         -4.80633620e+00,   3.34939231e+00,   1.32833665e+01,\n",
       "         -1.00807773e+01,  -2.78383183e+01,  -1.40784598e+01,\n",
       "         -6.25153134e+00,   6.46664344e+00,   3.97872941e+01,\n",
       "         -2.08249821e+01,  -2.20732092e+00,  -5.60823567e+00,\n",
       "         -1.30387341e+01,  -4.25301255e-01,  -4.94979837e+00],\n",
       "       [  5.31256628e+00,  -4.13417748e+00,  -1.99023934e+00,\n",
       "         -2.11138027e+00,  -2.74238172e+00,  -6.40307412e+00,\n",
       "         -5.17598706e+00,   9.03601350e+00,  -1.03513516e+01,\n",
       "         -4.79354918e+00,  -1.62966378e+01,  -7.17365637e+00,\n",
       "          8.99054408e+00,  -8.33241641e+00,  -1.46107611e+01,\n",
       "          1.08898453e+00,   3.01690039e+00,   8.80892075e-01,\n",
       "         -6.26260378e+00,  -4.48657668e+00,   2.12543750e+00,\n",
       "          4.42822357e+00,  -2.44426041e+01,  -8.82374902e+00,\n",
       "         -1.52875440e+00,  -3.33650276e+00,   2.63751988e+00,\n",
       "         -2.47231718e+00,   4.28986604e+00,   3.16551433e+01,\n",
       "          0.00000000e+00,  -1.10623705e-01,  -3.45867568e-01,\n",
       "         -2.31051798e+01,  -6.02717281e+00,   2.32642058e+01,\n",
       "         -8.00009952e+00,  -8.43061620e+00,  -1.46650304e+01,\n",
       "         -1.54381402e+01,   1.22460545e+01,  -1.01596564e+01,\n",
       "         -2.92938637e+01,   1.58430694e-01,  -5.34895737e+01,\n",
       "         -1.06570430e+01,  -2.10506384e+01,   1.46698843e+00,\n",
       "         -6.33422440e+00,  -1.23514413e+01,   1.74439959e+00,\n",
       "          1.08697452e+00,   1.14293874e+01,  -1.53171175e+00,\n",
       "         -2.85789761e+00,   5.52391222e+00,  -7.48962459e+00,\n",
       "         -1.13340786e+01,   3.06732470e+01,  -1.14489436e+01,\n",
       "         -8.20132675e+00,   1.82859663e+01,  -9.64949374e-01,\n",
       "         -1.63826221e+01,  -5.47282300e+00,   3.17834963e+00,\n",
       "          1.27043528e+01,  -7.20785613e+00,  -1.73861404e+01,\n",
       "         -5.53424191e+00,  -5.81169541e+00,  -6.04547977e+00,\n",
       "         -2.26453980e+01,   9.11870892e+00,  -1.27832606e+00,\n",
       "         -3.60131663e+00,   6.75577914e+00,  -1.25419581e+01,\n",
       "          8.41012535e-01,  -1.27177740e+00,  -8.99319772e+00,\n",
       "          5.96719443e+00,   9.85086548e+00,  -8.38845623e+01,\n",
       "         -2.71122794e+00,  -6.65325843e+00,   5.86061412e+00,\n",
       "         -1.20264416e+01,   8.68473709e+00,  -1.57246179e+01,\n",
       "          1.42313909e+01,   3.77096716e+00,  -1.17457229e+01,\n",
       "         -4.80720679e+01,  -4.16243238e+00,  -2.99146624e+00]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cPickle\n",
    "\n",
    "cPickle.dump(grid.best_estimator_, open(\"Otto_L2_log.pkl\", 'wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
